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Abstract 

The  concept  of  Unmanned  Aerial  Vehicles  (UAVs)  was  first  used  as  early  as  the 
American  Civil  War,  when  the  North  and  the  South  unsuccessfully  attempted  to  launch 
balloons  with  explosive  devices.  Since  the  American  Civil  War,  the  UAV  concept  has 
been  used  in  all  subsequent  military  operations.  Over  the  last  few  years,  there  has  been  an 
explosion  in  the  use  of  UAVs  in  military  operations,  as  well  as  civilian  and  commercial 
applications.  UAV  Mobile  Ad  Hoc  Networks  (MANETs)  are  fast  becoming  essential  to 
conducting  Network-Centric  Warfare  (NCW).  As  of  October  2006,  coalition  UAVs, 
exclusive  of  hand-launched  systems,  had  flown  almost  400,000  flight  hours  in  support  of 
Operations  Enduring  Freedom  and  Iraqi  Freedom  [1], 

This  study  develops  a  verified  network  model  that  emulates  UAV  network 
behavior  during  flight,  using  a  leading  simulation  tool.  A  flexible  modeling  and 
simulation  environment  is  developed  to  test  proposed  technologies  against  realistic 
mission  scenarios.  The  simulation  model  evaluation  is  performed  and  findings 
documented.  These  simulations  are  designed  to  understand  the  characteristics  and 
essential  performance  parameters  of  the  delivered  model.  A  statistical  analysis  is 
performed  to  explain  results  obtained,  and  identify  potential  performance  irregularities.  A 
systemic  approach  is  taken  during  the  preparation  and  execution  simulation  phases  to 
avoid  producing  misleading  results. 
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EVALUATION  OF  AN  OPNET  MODEL  FOR  UNMANNED  AERIAL 
VEHICLE  NETWORKS 

I  Introduction 


1.1.  Motivation 

The  Air  Force  is  interested  in  developing  a  verified  network  model  that  emulates 
Unmanned  Aerial  Vehicle  (UAV)  network  behavior  during  flight,  using  a  leading 
simulation  tool.  Through  simulation,  researchers  will  be  able  to  predict  the  expected 
performance  of  complex  wireless  networks.  A  flexible  modeling  and  simulation 
environment  is  needed  to  test  proposed  technologies  against  realistic  mission  scenarios 
to:  validate  architectures  and  topologies;  assess  protocols  and  solutions;  benchmark 
product  perfonnance  and  capabilities;  and  investigate  the  impact  of  changing  concept  of 
operations  on  mission  effectiveness  [2]. 

Traditionally,  network  simulation  researchers  use  simulator  to  evaluate  the  higher 
layers  of  the  Open  System  Interconnection  (OSI)  reference  model  [3,  4].  To  develop 
dependable  wireless  network  simulations,  researchers  must  focus  more  attention  on 
accurately  modeling  the  wireless  physical  and  medium-access-control  layers  of  the  OSI 
reference  model.  Specifically,  there  is  a  need  for  a  simulation  model  which  incorporates 
flight  mobility  characteristics,  antenna  characteristics,  radio  propagation,  signal 
interference,  and  standard  communication  protocols.  Ultimately,  there  is  a  great  desire  for 
a  simulation  environment  that  provides  a  capability  to  evaluate  several  aspects  of  UAV 
networks  through  simulation,  in  lieu  of  large  test-beds  or  costly  flight  testing. 


The  development  of  simulation  models  has  provided  a  strong  scientific 
contribution  to  the  computer  networking  field,  but  the  accuracy  and  credibility  of 
published  Mobile  Ad  Hoc  Network  (MANET)  simulation  research  have  come  under 
increasing  scrutiny  [5,  6,  7,  8].  Thus,  this  study  determines  the  simulation  parameters  that 
reflect  the  behavior  of  a  UAV  network  by  adhering  to  strict  development  and  validation 
procedures;  demonstrating  that  through  strict  validation  techniques,  a  credible  simulation 
model  can  be  achieved. 

1.2.  Overview  and  Goals 

The  goal  of  this  research  is  to  accurately  characterize  a  UAV  communication 
network  physical  layer  by  incorporating  key  parameters  such  as  antenna  radiation 
patterns,  signal  interference,  transmission  power,  data  rate,  and  flight  mobility  effects,  as 
well  as  validating  the  simulation  environment.  The  continuous  movement  of  a  UAV 
makes  it  challenging  to  accurately  model  this  highly  complex  communication  network  in 
a  simulation  model.  Therefore,  observation  data  collected  from  an  airborne  test-bed  was 
used  to  develop  the  simulation  model  in  this  study.  To  realistically  replicate  airborne 
MANETs  in  simulations  all  aspects  of  the  physical  layer  are  modeled.  This  study  also 
identifies  the  effects  of  inherent  limitations  for  a  UAV  network  that  may  impact  mission 
effectiveness. 

1.3.  Organization  and  Layout 

In  this  chapter,  the  motivations  for  the  research  were  discussed  along  with  an 
overview  and  goals  for  the  research.  Chapter  2  presents  background  material  related  to 


2 


MANET  simulation  development  and  execution,  and  provides  insight  into  recent 
published  MANET  research  studies.  Chapter  3  describes  the  methodology  used  for  the 
research  experiments.  Chapter  4  presents  the  data  and  analysis  from  this  research 
experiment.  Chapter  5  gives  an  overall  conclusion  of  the  research  perfonned. 
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II  Literature  Review 


2.1.  Chapter  Overview 

This  chapter  presents  background  material  relating  to  the  Mobile  Ad  Hoc 
Network  (MANET)  development  and  execution  in  simulation  environments.  The  first 
part  of  this  literature  review  presents  information  on  the  state  of  network  simulation 
relating  to  the  MANET  community  and  provides  insight  into  the  best  practices  or 
approaches  to  prepare  and  develop  a  simulation  that  produces  credible  results.  The 
second  part  of  this  literature  review  provides  infonnation  on  the  development,  execution, 
and  analysis  of  recent  airborne  MANET  simulations. 

2.2.  Current  State  of  MANET  Simulation  Research 

Computer  simulation  is  the  discipline  of  designing  models  of  theoretical  or  real 
physical  systems,  using  simulation  tools  for  evaluation  on  digital  computers.  The 
proliferation  of  computers  as  research  tools  has  resulted  in  the  adoption  of  computer 
simulation  as  one  of  the  most  commonly  used  paradigms  for  scientific  investigation  [5]. 
Much  of  the  knowledge  regarding  protocol  performance  for  wireless  networks  results 
from  computer  simulations  [6].  As  a  result,  the  development  of  simulation  models  has 
provided  a  strong  scientific  contribution  to  the  computer  networking  community. 
However,  the  accuracy  and  credibility  of  published  MANET  simulation  studies  have 
come  under  great  scrutiny. 

Early  efforts  to  alert  researchers  of  a  crisis  in  the  MANET  simulation  community 
took  place  in  May  1999.  National  Institute  of  Standards  and  Technology  and  US  Defense 
Advanced  Research  Projects  Agency  [9]  hosted  a  workshop  to  discuss  challenges  and 
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approaches  to  network  simulation.  At  that  time,  a  recent  paradigm  shift  to  using  network 
simulation  as  a  research  tool  was  the  reason  for  aforementioned  workshop.  Until  that 
time,  experimental  and  mathematical  models  were  the  primary  methods  used  in  early 
network  research.  With  networks  growing  in  complexity  due  to  the  mix  of  emerging 
wired  and  wireless  technology,  researchers  turned  to  simulation  as  a  means  to  understand 
complex  network  performance.  Heidemann  et  al.  [9]  took  an  early  look  at  how  simulation 
studies  were  being  validated.  Their  work  discussed  the  importance  of  network  researchers 
understanding  simulation  results  and  ensuring  that  the  validation  process  provides 
meaningful  answers  to  the  questions  being  researched. 

Pawlikowski  et  al.  [5]  conducted  one  of  the  first  studies  on  simulation  credibility 
for  telecommunication  networks.  They  investigated  computer  network  simulation  studies 
occurring  during  the  period  of  1992  to  1998.  Their  work  brought  awareness  to  the  fact 
that  “the  majority  of  the  published  simulation  results  lack  credibility  due  to  not  ensuring 
two  important  conditions:”  (1)  the  use  of  suitable  pseudo-random  number  generators 
(PRNG)  to  ensure  simulation  independence  and  (2)  the  appropriate  analysis  of  simulation 
output  data.  The  aforementioned  conditions  were  so  prevalent  that  not  all  network 
developers  and  users  were  enthusiastic  about  the  use  of  simulation  tools,  as  many  spoke 
of  a  deep  credibility  crisis. 

According  to  Pawlikowski  et  al.  [5],  a  “basic”  level  of  simulation  credibility  could 
be  achieved  if  the  following  reporting  guidelines  were  observed:  (1)  ensure  that  the 
reported  simulation  experiment  is  repeatable;  (2)  specify  the  analysis  method  of 
simulation  output;  and  (3)  specify  the  final  statistical  errors  associated  with  the  results. 
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Their  work  highlights  the  importance  of  ensuring  that  the  statistical  error  associated  with 
the  final  simulation  results  has  the  degree  of  confidence  within  the  accuracy  of  a  given 
confidence  interval.  Furthermore,  they  stress  simulation  experiments  should  be  controlled 
and  independently  repeatable. 

In  2003,  Perrone  et  al.  [6]  examined  the  state  of  the  MANET  simulation 
community  and  explored  how  adjusting  certain  parameters  may  affect  a  simulation’s 
accuracy.  Their  investigation  reiterated  the  importance  of  detail  required  to  conduct 
credible  simulation  studies.  They  emphasized  the  importance  of  following  well- 
established  simulation  techniques,  of  carefully  describing  simulation  scenarios  and 
parameters,  and  ensuring  that  the  underlining  simulation  assumptions  are  understood. 
Without  comprehensive  experimental  descriptions,  it  is  improbable  that  anybody  would 
be  capable  of  independently  repeating  or  building  upon  a  simulation  study.  Their  research 
highlighted  the  lack  of  rigor  and  detail  that  influences  simulation  results.  They  also 
concluded  that  the  vast  majority  of  MANET  simulation  studies  are  performed  using  a  few 
simulators,  namely  NS-2,  GloMoSim,  and  OPNET. 

The  2005  survey  [7]  over  the  2000-2004  ACM  International  Symposium  on 
Mobile  Ad  Hoc  Networking  and  Computing  (MobiHoc)  proceedings  exposed  significant 
credibility  shortfalls  within  published  simulation  studies.  From  their  observations,  the 
authors  proposed  that  simulation  credibility  is  contingent  on  the  following  conditions: 

1 .  Repeatable:  A  fellow  researcher  should  be  able  to  repeat  the  results  for  their  own 
satisfaction,  future  reviews,  or  further  development. 

2.  Unbiased:  The  results  must  not  be  specific  to  the  scenario  used  in  the  experiment. 
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3.  Rigorous:  The  scenarios  and  conditions  used  to  test  the  experiment  must  truly 
exercise  the  aspect  of  MANETs  being  studied. 

4.  Statistically  sound:  The  experiment  execution  and  analysis  must  be  based  on 
mathematical  principles. 

They  also  informed  researchers  about  the  following  simulation  pitfalls:  simulation 
'setup,  simulation  type,  model  validation  and  verification,  PRNG  validation  and 
verification,  variable  definition,  scenario  development,  simulation  execution,  setting  the 
PRNG  seed,  scenario  initialization,  metric  collection,  output  analysis,  single-set  of  data, 
statistical  analysis,  confidence  intervals,  and  publishing.  MANET  simulation-based 
research  is  an  involved  process  with  plenty  of  opportunities  to  compromise  the  study’s 
credibility. 

Andel  and  Yasinac  [8]  expressed  that  simulation  generalization  and  lack  of  rigor 
could  lead  to  inaccurate  data,  which  can  result  in  wrong  conclusions  or  inappropriate 
implementation  decisions.  They  emphasized  that  if  simulations  did  not  reflect  reality, 
they  cannot  give  insight  into  the  system  operating  characteristics  the  developers  are 
studying.  Table  1  outlines  their  identified  problems  and  recommendations  to  improving 
the  credibility  of  MANET  simulation. 
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Table  1:  MANET  Simulation  Problems  and  Recommendations 


PROBLEM 

SOLUTION 

Lack  of  independent 
repeatability 

Investigators  provide  all  settings  as  external  references  to 
research  Web  pages,  which  should  include  freely 
available  code/models  and  applicable  data  sets. 

Lack  of  statistical  validity 

Detennine  the  appropriate  number  of  required 
independent  simulation  runs,  addressing  sources  of 
randomness  that  may  affect  independent  runs. 

Use  of  inappropriate  models 

Use  two-ray  and  shadow  models  which  provide  a  more 
realistic  during  data  collection  and  analysis.  However, 
this  setting  should  be  validated  against  some  baseline 
data.  This  advice  relates  to  the  next  point. 

Improper  or  nonexistence  of 
simulation  validation 

Validate  the  complete  simulation  against  real-world 
implementation  to  mitigate  the  aforementioned  problem. 

Unrealistic  application  traffic 

Create  nodes  with  real-world  characteristics 

Improper  precision 

Use  MANET  simulations  to  provide  proof  of  concept  and 
general  performance  characteristics,  not  to  directly 
compare  multiple  protocols  against  one  another. 

Lack  of  sensitivity  analysis 

Sensitivity  analysis  can  identify  a  chosen  factor’s 
significance.  That  is,  the  root  cause  of  measurement 
deltas,  must  be  fully  attributable  to  an  underlying  factor. 

For  example,  is  the  difference  between  two  simulated 
protocols  due  to  the  protocols  or  could  it  be  due  to  the 
underlying  settings? 

2.2.1.  Simulations  Factors 

As  stated  by  numerous  researchers,  developing  a  credible  simulation  requires  the 
appropriate  level  of  detail  and  rigor.  The  following  paragraphs  outline  various  items  to 
consider  when  conducting  MANET  simulations. 

2. 2. 1.1.  Simulators 

There  are  a  number  of  simulators  available  to  the  MANET  community  to  conduct 
network  research.  Table  2  provides  insight  into  the  popularity  of  simulators  utilized  in 
2005. 
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Table  2:  Popularity  of  Simulations,  2005  [10] 


NAME 

POPULARITY 

LICENSE 

NS-2 

88.8% 

Open  Source 

GloMoSim 

4% 

Open  Source 

OPNET 

2.61% 

Commercial 

QualNet 

2.61% 

Commercial 

OMNet++ 

1.04% 

Free  for  academic  and  educational  use 

NAB 

0.48% 

Open  Source 

J-Sim 

0.45% 

Open  Source 

SWANS 

0.3% 

Open  Source 

GTNets 

0.13% 

Open  Source 

Pdns 

<0.1% 

Open  Source 

DIANEmu 

<0.1% 

Free 

Jane 

<0.1% 

Free 

Regardless  of  the  chosen  simulator,  a  simulator  can  only  be  characterized  as 

dependable  and  realistic;  no  network  simulator  can  be  described  as  accurate  [10].  Calvin 

et  al.  [11]  conducted  an  experience  on  the  accuracy  of  MANET  simulators.  Their 

findings  showed  that  there  exists  significant  divergence  between  the  leading  simulators: 

OPNET,  NS-2,  and  GloMoSim.  If  a  simulator  is  valid,  real-life  performance  should 

correlate  with  the  simulated  performance  [8]. 

2.2. 1 .2.  Simulation  Type  and  Objective 

The  2005  survey  by  Kurkowski  et  al.  [7]  showed  that  57.9  percent  of  the 

publications  they  reviewed  did  not  state  the  simulation  type  that  was  being  perfonned. 

This  apparently  minor  step  could  lead  to  a  miscalculation  in  simulation  results.  The  two 

simulation  types  that  are  central  to  computer  network  research  are:  steady-state  and 

terminating  simulation.  In  tenninating  simulations,  the  simulation  has  a  specific  starting 

and  stopping  condition,  with  a  well-defined  run-time.  Whereas  in  steady-state 

simulations,  the  initial  conditions  do  not  matter  and  it  is  not  important  how  the  simulation 
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terminates.  The  focus  of  steady-state  simulations  is  to  study  a  condition  in  which  some 
specified  characteristic  of  a  condition,  such  as  a  value,  rate,  periodicity,  or  amplitude, 
exhibits  only  negligible  change  over  an  arbitrarily  long  period.  A  steady-state  can  be 
accomplished  by  simulation  wann-up.  The  major  reason  for  the  failure  of  many  network 
simulations  is  due  to  the  lack  of  clearly  understanding  the  research  goal,  and  ensuring  all 
objectives  are  attainable  [12]. 

2.2. 1.3.  Simulation  Size 

Simulation  users  need  to  understand  both  what  is  provided  in  a  simulator  and 
what  is  appropriate  for  their  experiment  [9].  Riley  et  al.  [13]  explained  that  there  exists  a 
threshold  on  the  number  of  nodes  in  a  network  for  which  the  results  obtained  no  longer 
vary  as  the  number  of  nodes  increases. 

2.2.1 .4.  Simulation  Wann-up 

Research  studies  by  [6,  7]  highlighted  that  researchers  tend  to  pay  little  attention 
to  the  fact  that  one  or  more  of  their  sub-models  may  require  initialization  or  a  wann-up 
time  to  avoid  bias  in  their  simulation’s  performance.  Detennining  and  reaching  the 
steady-state  level  of  activity  is  part  of  the  initialization,  which  must  be  performed  prior  to 
data  collection.  Data  generated  prior  to  reaching  steady-state  is  biased  by  the  initial 
simulation  conditions  and  cannot  be  used  in  the  analysis  [7].  The  case  study  written  by 
Penone  et  al.  [6]  illustrates  how  the  random  waypoint  mobility  (RWM)  model  could 
cause  considerable  errors  to  a  simulation  if  initialization  is  not  considered.  The  RWM  is 
based  on  the  following  three  parameters:  pause  time,  min  speed,  and  max  speed.  All 
mobile  nodes  start  out  paused  and  begin  to  move  at  the  same  time:  the  end  of  the  initial 
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pause.  Their  work  revealed  the  fact  that  for  the  RWM  the  level  of  mobility  goes  through 
oscillations  before  settling  down  onto  a  steady  state.  The  classic  solution  for  this  effect  is 
the  application  of  data  deletion. 

2.2. 1.5.  Mobility  Models 

It  is  critical  for  network  simulation  mobility  models  to  match  real-world 
parameters  to  ensure  that  simulation  results  are  meaningful  [11].  Mobility  models  are 
used  to  define  the  node  movement  in  MANET  simulations.  These  models  fall  into  two 
categories:  independent  and  group-based  models.  In  independent  models,  the  movement 
of  each  node  is  modeled  autonomously  from  other  nodes  in  the  simulation.  In  group 
mobility  models,  there  is  some  association  among  the  nodes  and  their  movements 
throughout  the  cells  or  simulation  area.  Traces  and  synthetic  models  are  two  types  of 
mobility  models  used  in  MANET  simulation  [14].  In  the  traces  models,  mobility  patterns 
are  observed  in  real-life  systems  and  imported  into  the  simulation.  However,  new 
network  environments  (e.g.,  ad  hoc  networks)  are  not  easily  modeled  if  traces  have  not 
yet  been  created.  In  this  type  of  situation  it  is  necessary  to  use  synthetic  mobility  models. 
These  models  attempt  to  realistically  represent  the  behaviors  of  mobile  nodes  without 
pre-observed  traces.  Previously,  researchers  relied  on  randomized  mobility  models,  most 
commonly  on  the  RWM  models  [10].  Network  researchers  are  well  aware  of  the  negative 
impact  that  RWM  models  have  on  simulation  accuracy.  These  models  are  idealistic  rather 
than  realistic,  because  a  real-world  host  will  not  move  randomly  without  any  destination 
point  [15]. 
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2.2.1 .6.  Radio  Wave  Propagation  Model 

Radio  propagation  is  expensive  to  model  (in  both  development  and  run-time)  and 
difficult  to  abstract  [9].  Researchers  are  increasingly  aware  of  the  need  to  develop  radio 
propagation  models  which  include  more  realistic  features,  such  as  hills,  obstacles,  link 
asymmetries,  and  unpredictable  fading.  Many  widely  used  models  embody  the  following 
set  of  assumptions:  the  world  is  two  dimensional;  a  radio’s  transmission  area  is  roughly 
circular;  all  radios  have  equal  range;  if  I  can  hear  you,  you  can  hear  me;  if  I  can  hear  you 
at  all,  I  can  hear  you  perfectly;  and  signal  strength  is  a  simple  function  of  distance  [16], 

The  two  leading  network  simulators  have  varying  radio  propagation  models.  The 
NS-2  network  simulator  has  four  frequently  used  models:  the  Free  Space  Model,  Two- 
Ray  Ground  Model,  Ricean  and  Rayleigh  Fading  Models,  and  Shadowing  models. 
Whereas,  the  OPNET  network  simulator  has  the  following  frequently  used  models: 
CCIR,  Free  Space  Model,  Hata  Model,  Longley-Rice  Model,  Terrain  Integrated  Rough 
Earth  Model,  and  Walffish-Ikegami  Model.  More  realistic  models  take  into  account 
antenna  height  and  orientation,  terrain  and  obstacles,  surface  reflection  and  absorption, 
and  so  forth.  Simplistic  models  can  dramatically  affect  simulation  results  [16]. 

2.2.1 .7.  Routing  Protocols 

The  issues  of  determining  viable  routing  paths  and  delivering  messages  in 
MANETs  are  well-documented  problems.  Factors  such  as  fluctuating  wireless  link 
quality,  propagation  path  loss,  interference,  signal  fading,  varying  topological  changes 
and  power  consumption  affects  a  MANET  routing  protocol’s  ability  to  provide  a  reliable 
communication  platform.  Generally,  two  classes  of  routing  protocols  have  been  designed 
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for  MANETs,  a  proactive  set  of  routing  protocols  (e.g.,  Optimized  Link  State  Routing 
protocol)  and  a  reactive  set  of  routing  protocols  (e.g.,  Ad-hoc  On-demand  Distance 
Vector  (AODV)  protocol)  [17].  Chin  et  al.  [18,  19]  looked  at  the  implementation  of  two 
distance  vector  routing  protocols  using  an  operational  ad-hoc  network.  During  the  course 
of  their  experiment  they  discovered  a  number  of  problems  with  both  protocols.  They  [19] 
highlighted  the  following  four  issues  for  further  research: 

1 .  Handling  unreliable/unstable  links 

2.  Minimizing  the  dependency  on  topology  specific  parameters 

3.  Mechanisms  for  handoff  and  reducing  packet  loss  during  handoff 

4.  Incorporating  neighbor  discovery  and  filtering  into  neighbor  selection  sub¬ 
layer 

Previously,  research  has  demonstrated  that  routing  protocols  significantly  impact 
the  simulation  outcome.  For  the  simulation  to  be  constructive,  investigators  must  clearly 
understand  and  document  their  setting  choices  within  the  respective  simulation  tool  [8]. 

2.2. 1 .8.  Simulation  Randomness 

Numerous  researchers  have  emphasized  the  MANET  community’s  lack  of 

appropriately  selecting  a  PRNG.  Kurkowski’s  et  al.  [7]  MobiHoc  publication  survey 

revealed  that  none  of  the  84  simulation  papers  they  reviewed  (publications  that 

mentioned  PRNG)  addressed  PRNG  issues.  Pawlikowski  et  al.  [5]  recommended 

addressing  the  issues  associated  with  the  improper  use  of  PRNG  by:  ensuring  that 

simulations  used  a  PRNG  that  is  appropriate  for  the  simulation.  For  example,  use  a 

PRNG  with  adequately  long  cycles  that  can  be  used  in  more  than  one  simulation;  and 

using  an  established  PRNG  that  has  been  tested  thoroughly.  In  the  case  when  using  a 
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simulator  that  has  an  internal  PRNG,  such  as  OPNET  and  NS-2,  researchers  should 
ensure  that  the  seed  of  PRNG  is  set  correctly  for  each  simulation  run. 

2.2. 1 .9.  Simulation  Validation 

Proper  validation  provides  confidence  that  your  simulation  tool,  environment,  and 
assumptions  do  not  alter  the  answers  to  the  questions  being  analyzed.  MANET 
simulation  studies  pose  several  challenges  to  modeling,  such  as  addressing  fading, 
interference,  and  mobility  effects.  These  factors  can  produce  significant  impact  to  results 
in  the  modeled  wireless  network  expected  and  observed  performance.  Surprisingly,  there 
are  no  widely  accepted  practices  that  exist  to  help  validate  and  evaluate  trustworthiness 
of  simulation  results.  Heidemann  et  al.  [9]  suggests  evaluating  simulation  sensitivity  to 
help  understand  how  varying  configurations  change  a  simulation’s  accuracy. 

2.2. 1.10.  Simulation  Data  Analysis 

Pawlikowski  et  al.  [5]  stressed  the  importance  of  conducting  the  appropriate 
analysis  of  the  simulation  output  results.  The  authors  emphasize  ensuring  that  the 
statistical  error  associated  with  the  final  results  has  the  degree  of  confidence  in  the 
accuracy  of  a  given  confidence  interval.  Non-rigorous  output  analysis  leads  to  the 
inaccuracy  of  many  simulation  studies.  Kurkowski  et  al.  [7]  showed  that  only  12  percent 
of  MobiHoc  simulation  results  appear  to  be  based  on  sound  statistical  techniques. 

2.3.  Related  Work 

A  MANET  is  a  collection  of  mobile  nodes  that  operate  autonomously  among 
themselves  over  dynamic  wireless  interfaces.  These  nodes  usually  communicate  over 
bandwidth  constrained  wireless  links.  Figure  1  shows  a  fixed  wired  network  connected 
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through  a  cable  or  fiber  backbone  link,  while  a  MANET  is  dynamically  interconnected  by 
a  wireless  link  that  does  not  require  an  access  point.  There  are  several  advantages  with 
implementing  a  MANET,  the  most  obvious  advantage  is  mobility.  Mobility  provides  a 
great  deal  of  flexibility,  which  can  translate  into  rapid  network  deployment,  providing  an 
extremely  favorable  solution  to  military  applications.  The  Department  of  Defense’s  [20] 
Network-Centric  Warfare  (NCW)  initiative  is  to  develop  and  leverage  infonnation 
superiority  that  generates  increased  combat  power  by  networking  sensors,  decision¬ 
makers,  and  shooters  to  achieve  shared  awareness,  increased  speed  of  command,  higher 
tempo  of  operations,  greater  lethality,  increased  survivability,  and  a  degree  of  self¬ 
synchronization. 


Traditional  Fixed-Wired  Topology 

,v. 

MANET  Topology 

Network  Node  ■  Access  Point —  Cable/Fiber  Backbone  . Wireless  Link 

Figure  1:  Fixed  Wired  Network  versus  MANET 

The  attraction  to  MANET  technology  stems  from  the  ease  in  which  these 
networks  can  effectively  link  several  dispersed  entities  with  one  another,  without 
requiring  a  fixed  infrastructure.  MANET  technology  provides  a  means  to  accommodate  a 
diverse  mix  of  platfonns  and  systems  which  is  critical  to  the  success  of  military 
operations  [21].  In  2006,  the  United  States  utilized  UAVs  to  fly  almost  400,000  hours  in 
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support  of  Operation  Enduring  Freedom  and  Iraqi  Freedom.  These  UAVs  were  equipped 
with  wireless  transmitters  and  receivers  using  antennas  to  exchange  information  with 
other  entities. 

Despite  the  fact  that  MANETs  [1]  continue  to  prove  their  worth  in  military 
operations,  there  are  several  limitations  associated  with  MANET  technology.  Many 
nodes  (e.g.,  micro-UAVs)  in  a  MANET  are  reliant  on  batteries  or  other  exhaustible 
means  for  their  energy,  which  impact  or  restrict  the  time  a  node  is  able  to  function  in  an 
area  of  operation.  Another  major  constraint  is  limited  physical  security.  Mobile  nodes  are 
susceptible  to  jamming,  eavesdropping,  spoofing,  denial-of-service  attacks,  and  possible 
physical  capture.  These  threats  are  often  mitigated  by  applying  various  encryption  and 
security  techniques.  But,  security  comes  at  a  cost  to  throughput  and  efficiency.  These 
networks  are  not  usually  built  with  security  protocols  in  mind,  but  as  an  afterthought  once 
vulnerabilities  have  been  identified  [22],  MANET  link  reliability  continues  to  be  a  major 
concern,  since  these  networks  are  based  on  radio  waves,  the  network  behavior  can  be 
somewhat  unpredictable  due  to  propagation  problems  that  may  interrupt  the  radio  link. 
Even  with  these  limitations,  MANETs  remain  a  popular  choice  for  military  applications. 

2.3.1.  Traditional  MANET  Research 

Chin  et  al.  [19]  reported  on  their  experience  of  building  an  operational  ad-hoc 
network  that  transmitted  useful  data.  The  authors’  study  differs  from  previous  studies  in 
the  fact  that  their  work  focused  on  the  operational  feasibility  of  existing  routing  protocols 
and  the  effort  to  create  a  reliable  ad-hoc  network.  They  examined  two  distance  vector 
MANET  routing  protocols,  Ad  Hoc  On-Demand  Distance  Vector  (AODV)  and 
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Destination-Sequenced  Distance  Vector  (DSDV).  By  conducting  experiments  on  a  test¬ 
bed  consisting  of  two  notebooks  and  three  desktop  computers,  they  were  able  to  show 
that  neither  protocol  could  provide  a  stable  route  over  any  multi-hop  network  connection. 
Each  protocol  was  fooled  by  the  transient  availability  of  the  network  links  to  nodes  that 
were  more  than  one  hop  away.  A  fading  channel  caused  the  routing  protocols  to  conclude 
incorrectly  that  there  was  a  new  one  hop  neighbor.  The  results  from  their  research  test¬ 
bed  versus  the  results  from  a  simulation  environment  differed.  The  simulation  application 
they  utilized  provided  an  inaccurate  assessment  of  actual  protocol  perfonnance.  They 
suspected  that  the  results  dissimilarity  was  due  to  the  use  of  a  simplistic  radio 
propagation  model  that  was  standard  in  the  simulation  package.  They  recommended 
using  realistic  radio  propagation  models  that  incorporated  channel  fading  and  other 
important  wireless  channel  characteristics.  This  thesis  examines  the  OPNET  radio 
propagation  model  to  ensure  that  the  model  adequately  supports  our  UAV  network.  This 
objective  is  accomplished  by  comparing  our  simulation  results  with  actual  flight  test 
performance  metrics  in  order  to  validate  the  model. 

2.3.2.  Airborne  MANET  Studies 

Preston  et  al.  [23]  used  real-world  data  to  look  at  the  quality  of  service  over 
airborne  radio  links  which  experienced  periodic  outages  due  to  line  of  sight  occlusion 
caused  by  the  aircraft’s  wings  and  tail.  Their  study  extended  the  standard  OPNET 
models  so  that  the  pointing  direction  of  an  antenna  affixed  to  a  moving  aircraft  could  be 
determined  in  three-dimensional  space.  OPNET  Version  10.5  and  earlier  did  not  support 
node  mobility  modeling  with  six  degrees  of  freedom.  Earlier  versions  only  provided 
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mobile  node  position  in  three  degrees  of  freedom:  latitude,  longitude,  and  altitude. 
OPNET  Version  11.0  and  later  includes  three  additional  degrees  of  freedom:  roll,  pitch, 
and  yaw.  However,  initially  the  standard  OPNET  pipeline  stages  did  not  take  advantage 
of  these  new  degrees  of  freedom.  Due  to  this  fact,  Preston  et  al.  modified  a  user  supplied 
“Enhanced  Antenna  Positioning”  model  to  include  all  six  degrees  of  freedom.  In 
addition,  they  also  modified  the  OPNET  receive  and  transmit  antenna  gain  pipeline 
stages  to  incorporate  all  six  degrees  of  freedom  in  order  to  describe  the  motion  of  an 
antenna  mounted  to  an  aircraft  in  flight.  The  radio  transceiver  pipeline  stages  in  Figure  2 
consists  of  14  stages  that  exhibit  the  radio  link  behavior  perfonning  all  the  wireless 
physical  layer  operations.  Each  pipeline  stage  can  be  modified  or  substituted  to  fit  the 
experimental  goals. 

According  to  Law  and  Kelton  [24],  the  most  definitive  test  of  a  simulation 
model’s  validity  is  establishing  that  its  output  data  closely  correlate  to  the  output  data  that 
would  be  expected  from  the  actual  system.  To  validate  these  enhanced  OPNET  models, 
Preston  et  al.  [23]  designed  a  scenario  based  on  communication  between  the  United 
States  Air  Force’s  Paul  Revere  test  aircraft  and  a  ground  station.  Their  research  clearly 
showed  that  as  the  aircraft  changed  altitude  and  position,  so  should  the  antenna  pointing 
direction,  to  obtain  accurate  simulation  results. 

Their  experiment  showed  a  good  correlation  between  the  OPNET  model  they 
developed  and  the  Paul  Revere  flight  test  data,  but  there  were  instances  in  the  simulation 
that  did  not  match  the  flight  data.  These  anomalies  were  the  result  of  an  imprecise 
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antenna,  and  the  fact  that  the  antenna  pattern  in  the  simulation  study  did  not  match  the 
Paul  Revere  aircraft  antenna. 


Figure  2:  OPNET  Transceiver  Pipeline  Stages  [25] 

This  thesis  intends  to  follow  a  similar  approach,  but  we  conduct  multiple 

simulations  comparing  communication  performance  based  on  three  sets  of  test  data.  This 

approach  supports  our  claim  that  airborne  MANET  simulations  must  incorporate  real 

trace  models,  as  well  as  incorporating  the  suitable  level  of  detail  in  model  development. 

Denson  et  al.  [26]  modeled  the  performance  of  MANETs  with  random  and 

predetermined  mobility  patterns  using  OPNET.  They  analyzed  how  well  sensor  nodes 

were  able  to  fonn  a  cluster,  and  maintain  a  formation  with  varying  update  intervals 

between  the  nodes  and  the  mobile  base  station.  Their  research  relates  closely  to  NCW 
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scenarios,  where  sensor  nodes  are  deployed  randomly  from  an  aircraft  onto  the  area  of 
operation  with  the  expectation  that  a  cluster  would  be  formed  to  collect  infonnation  on 
the  environment  or  on  adversary  troop  movement.  The  authors  implemented  a  scenario 
consisting  of  four  mobile  nodes  and  a  signal  mobile  base  station  to  show  the  effect  of 
reference  point  broadcast  interval  on  the  position  error  of  the  mobile  nodes  and  the 
mobile  base  station  power  consumption  per  packet  transmission.  But,  they  had  to  specify 
three  additional  attributes  in  the  standard  OPNET  manet_station_adv  model  to 
characterize  the  behavior  of  their  scenario.  These  attributes  are  defined  as: 

•  Movement  Pattern:  defines  a  node  to  be  either  a  base  station,  or  a  simple  mobile 
node 

•  Follow  Target:  defines  which  group  the  node  belongs  to,  in  the  case  where  a 
node  is  not  a  base  station 

•  Follow  Distance:  defines  the  distance  in  meters  which  the  mobile  nodes  should 
maintain  from  the  central  base  station. 

Their  research  demonstrates  the  flexibility  of  the  OPNET  simulation  package  to 

model  complex  scenarios  that  would  otherwise  be  costly  to  execute  in  real-world  test 

environments.  Our  research  provides  another  example  of  how  OPNET  models  must  be 

modified  or  substituted  to  emulate  a  real-world  system.  Proving  it  is  essential  to  examine 

standard  OPNET  models  to  ensure  that  these  models  meet  the  experiment  requirements. 

To  support  their  Unmanned  System  Initiative,  the  US  Anny  partnered  with 

Auburn  University  to  develop  a  high  fidelity  modeling  and  simulation  test-bed  for  Army 

UAVs  [27].  The  control  station  that  the  soldiers  used  to  communicate  with  a  UAV  was 

required  to  connect  to  a  base  station  antenna  on  the  ground  utilizing  over  400  feet  of 

various  cables.  This  setup  presented  a  major  problem  in  that  it  was  time  consuming  to  set 
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the  system  up  and  take  down.  The  large  radio  footprint  presented  a  potential  risk  of  the 
enemy  determining  the  base  station  location  through  RF  triangulation.  The  Army 
addressed  this  problem  by  developing  a  simulation  test-bed  to  evaluate  secure  wireless 
alternatives  to  replace  the  troublesome  setup.  The  Anny  had  the  need  to  do  verification 
and  validation  (V&V)  to  ensure  that  the  test-bed  had  appropriate  predictive  power. 
Through  proper  V&V,  the  simulation  or  test-bed  can  be  used  to  test  various  network 
configurations.  They  used  parts  of  actual  field  data  as  a  means  to  build  their  test-bed,  and 
the  remaining  data  to  detennine  whether  the  model  behaves  as  the  system  does.  The 
author  points  out  that  one  must  be  concerned  that  conceptual  models  correctly  abstract 
the  unimportant  details  while  still  capturing  the  attributes  that  drive  the  simulation.  This 
thesis  utilizes  the  simplified  version  of  modeling  process  outlined  in  Figure  3  which 
depicts  the  simulation  V&V  process. 


VERIFICATION 


Figure  3:  Simplified  version  of  the  modeling  process  [28] 
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They  [27]  also  calibrated  their  model  by  using  input  scripting.  They  took  actual 
field  test  data  and  configured  the  simulator  to  read  the  inputs.  They  then  used  the 
simulation  test-bed  to  evaluate  potential  designs  to  solve  their  problem.  Their  study 
successfully  followed  four  commonly  accepted  scientific  method  steps  outlined  by  [29]: 

1 .  Ability  to  observation  of  the  system 

2.  Ability  to  account  for  observed  behavior 

3.  Ability  to  predict  future  behavior  based  on  assumption  that  the  modeling 
understanding  is  correct 

4.  Ability  to  compare  the  predicted  behavior  with  actual  behavior 

In  addition,  they  made  modifications  to  the  802. 1 1  g  wireless  model  in  OPNET  to 
simulate  the  transmission  of  UAV  data  over  a  wireless  network.  The  result  of  their 
research  was  UAV  simulations  that  furthered  the  goal  of  making  UAV  command  stations 
more  mobile.  The  author  has  plans  to  extend  this  model  in  both  Qualnet  and  NS-2. 

The  military  is  constantly  searching  for  ways  to  enhance  Network-Centric 
Warfare.  Airborne  networks  consisting  of  command  and  control  aircrafts  such  as  the 
Airborne  Warning  and  Control  System,  Rivet  Joint,  Joint  Surveillance  Target  Attack 
Radar  System,  and  UAVs  are  critical  to  NCW  objectives.  An  airborne  network  often 
consists  of  high-bandwidth  links  that  periodically  suffer  outages  at  predicable  times  due 
to  aircraft  banking  (e.g.,  roll,  pitch,  and  yaw)  in  flight  profiles  [30].  Butler  et  al. 
investigated  a  methodology  for  emulating  MANETs  during  a  flight  test  utilizing  an 
airborne  network  based  on  wide-body  aircraft.  Figure  4  shows  the  testing  architecture 


22 


used  in  their  experiment  which  is  comprised  of  a  single  aircraft,  two  simulated  airborne 
nodes,  and  two  emulated  MANETs. 


Airborne  Network 


MANET  1 


MANET  2 


Figure  4:  Interconnected  MANETS  [30] 

Bulter  et  al.  [30]  designed  each  MANET  node  to  consist,  at  a  minimum,  of  a  radio 
and  a  router.  In  addition,  some  nodes  included  a  host,  or  hosts,  and  a  gateway.  In  the 
case  when  nodes  were  not  in  direct  radio  contact  with  each  other  or  with  MANET  nodes 
serving  as  gateway,  a  router  function  was  included  to  allow  nodes  to  communicate 
beyond  a  single  hop.  They  were  able  to  emulate  the  MANET  networks  by  making  a 
series  of  simplifying  design  choices  and  assumptions.  These  assumptions  and  design 
choices  can  be  found  in  [30].  The  scenario  consisted  of  two  MANETs  in  separate 
locations  with  nodes  traversing  the  network.  When  a  node  entered,  a  MANET  a  route  was 
created  and  added  to  the  routing  table.  In  addition,  they  created  an  application  to  collect 
and  record  all  of  the  data  entering  and  leaving  the  MANET  during  the  flight  test.  Their 
research  identified  several  fundamental  issues  and  problems  involved  with  supporting  the 
MANETs  over  the  Airborne  Network.  Specifically,  they  noted  that  the  aircraft  flight 
profile  during  banks  to  turn  blocked  the  transmission  signal.  Regardless  of  whether  the 
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antennas  were  Omni-directional  or  directional,  mounted  in  the  nose,  belly,  or  on  the  roof 
of  the  aircraft,  some  blocking  by  the  aircraft  body  was  likely  to  occur.  It  is  possible  to 
minimize  the  loss  of  connectivity  by  switching  between  multiple  antennas  on  board  the 
aircraft  to  create  a  new  connection  as  the  aircraft  proceeds  through  the  flight  profile  [30]. 
In  addition,  Swanson  [31]  conducted  a  recent  experiment  involving  airborne  networks. 
His  study  was  able  to  show  that  the  OPNET  model  developed  for  the  Tactical  Targeting 
Network  Technology  (TTNT)  Airborne  Network  accurately  represented  the  real-world 
concept.  The  systematic  approach  used  to  conduct  the  research  was  essential  to 
accurately  validate  and  verify  the  TTNT  OPNET  model. 

2.4.  Wireless  Networking 

Wireless  networking  involves  getting  infonnation  from  one  location  to  another 
location  using  electromagnetic  waves,  such  as  radio  waves.  Wireless  telecommunication 
has  a  significant  impact  on  the  way  the  United  States  military  conducts  combat 
operations.  There  is  no  aspect  of  military  operations  that  wireless  communication  does 
not  support,  ranging  from  using  cellular  phones  to  a  fully  functional  airborne  MANET; 
wireless  communication  enhances  the  combat  operator’s  to  ability  share  infonnation. 
Emerging  wireless  communication  technologies  have  lead  to  wired  networks  being 
replaced  by  more  flexible  wireless  networks  at  an  exponential  rate.  These  technologies 
address  the  military  need  to  have  information  accessible  to  all  elements  of  the  force  at 
any  time  and  any  place  through  the  use  of  manned  and  unmanned  vehicles.  Figure  5 
illustrates  the  Global  Infonnation  Grid  (GIG)  concept  adopted  by  the  Department  of 
Defense  in  2002.  Wireless  networking  continues  to  be  an  essential  component  to 
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successfully  implementing  the  GIG  objective  of  providing  authorized  users  with  a 
seamless,  secure,  and  interconnected  information  environment,  meeting  real-time  and 
near  real-time  needs  of  the  warfighter  [32]. 


Figure  5:  Global  Information  Grid  [32] 


In  order  for  wireless  communication  to  provide  military  forces  with  a  seamless 
information  environment,  several  technical  challenges  must  be  addressed,  such  as 
network  unpredictability,  bandwidth  and  power  limitations,  security,  latency,  availability, 
and  link  quality.  Modeling  and  simulation  research  is  instrumental  in  solving  many  of  the 
above  issues. 

2.4.1.  Physical  Layer  Protocol 

The  OSI  Physical  layer  is  responsible  for  the  transmission  of  raw  bits  over  the 
physical  link  connecting  network  nodes.  A  transmission  link  can  consist  of  either  a  wired 
or  wireless  medium.  This  layer  includes  specifications  for  electrical  and  mechanical 
characteristics  such  as:  signal  timing,  voltage  levels,  data  rate,  maximum  transmission 
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length,  and  physical  connectors  of  networking  equipment.  Figure  6  shows  the  seven  layer 
OSI  model  which  depicts  how  network  protocols  and  equipment  interact  and 
communicate  with  each  other  [33,  34]. 


The  physical  layer  of  the  OSI  model  is  the  first  layer  of  the  model.  Its  purpose  is 
to  define  the  relationship  between  a  device  (adapters,  router,  hubs,  etc.)  and  a  physical 
medium  (guided  or  unguided).  In  wireless  networks,  nodes  usually  use  radio  frequency 
channels  as  their  physical  medium.  Since  the  nodes  in  a  wireless  network  are  not 
physically  connected,  there  is  a  great  deal  of  flexibility  with  implementing  a  wireless 
network. 

As  stated  by  a  previous  MANET  study  [35],  characterizing  the  physical  layer 
brings  many  challenges  due  to  the  complexity  and  unpredictability  associated  with 
wireless  communication  links.  Wired  network  models  have  advanced  to  a  degree  that 
researchers  understand  the  physical  layer  parameters  that  significantly  affect  the  accuracy 
of  simulation  results.  As  a  result,  suitable  wired  abstractions  have  been  developed.  While 
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in  wireless  networks  there  is  less  information  regarding  the  appropriate  level  of  detail 
required  to  ensure  the  correctness  of  network  simulations. 

Heidemann  et  al.  [3]  describe  the  trade-offs  associated  with  adding  detail  to 
simulation  models.  They  looked  at  the  effects  of  detail  in  five  case  studies  of  wireless 
simulations  for  protocol  design.  They  pointed  out  that  too  little  detail  can  produce 
simulations  that  are  misleading  or  incorrect,  while  adding  detail  requires  time  to 
implement,  debug,  and  later  change.  A  side-effect  of  adding  detail  is  that  it  slows  down 
the  simulation,  and  can  distract  from  answering  the  intended  research  question.  They 
stated  that  a  “fully  realistic”  simulation  is  not  possible,  and  the  challenge  to  simulation 
designers  is  to  identify  what  level  of  detail  is  required  to  answer  the  design  questions  at 
hand.  Consequently,  choosing  the  right  level  of  detail  for  a  wireless  network  simulation  is 
not  a  trivial  task. 

Takai  et  al.  [4]  point  out  that  researchers  traditionally  develop  simulation  models 
that  are  used  to  evaluate  devices  and  protocols,  such  simulations  usually  focus  on  higher 
layers  OSI  reference  model  (e.g.,  network,  transport).  They  present  several  factors  at  the 
physical  layer  that  are  relevant  to  the  performance  evaluations  of  higher  layer  protocols. 
These  factors  include  signal  reception,  path  loss,  fading,  interference  and  noise 
computation,  and  preamble  length.  Modeling  the  physical  layer  for  our  research  requires 
sensitivity  analysis  of  the  aforementioned  factors  to  accurately  reproduce  the  system 
under  evaluation. 
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2.4.2.  Antenna  Characterization 

Antennas  have  become  an  indispensable  component  of  military  communication 
infrastructure.  The  antenna  selection  is  one  of  the  most  important  components  in  any 
radio  communication  system.  As  a  result,  properly  defining  the  antenna  radiation  pattern 
of  a  mobile  node  is  essential  to  replicating  the  network  in  a  simulation.  Two  very 
important  parameters  related  to  the  design  of  antennas  are  gain  and  directivity. 

Antenna  gain  is  the  measure  in  decibels  how  much  more  power  an  antenna 
radiates  in  a  certain  direction  with  respect  to  a  hypothetical  ideal  isotropic  antenna,  which 
radiates  equally  in  all  directions.  Thus,  gain  is  calculated  by  using: 

P  (AUT) 

G  = - — - - x  G(isotropic  antenna)  (1) 

Pmax  (isotropic  antenna) 

where  Pmax  (AUT)  is  the  maximum  power  density  of  the  Antenna  Under  Test  (AUT), 
Pmax  (isotropic  antenna)  is  the  maximum  power  density  of  the  ideal  reference,  and 
G(isotropic  antenna)  is  the  known  gain  of  the  ideal  reference  antenna.  Directivity  is  equal 
to  the  ratio  of  the  maximum  power  density  P(0,(p)max  (watts/m  )  to  its  average  value  over 
a  sphere  as  observed  in  the  far  field  of  an  antenna.  Thus,  directivity  from  pattern  is 
calculated  by  using: 

n  _ 

(2) 

Non-isotropic  antennas  are  characterized  by  how  much  more  intensely  the 
antenna  radiates  in  its  preferred  direction  than  an  ideal  reference  antenna  would  when 
transmitting  at  the  same  total  power  [36].  Figure  7  demonstrates  two  types  of  antenna 
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patterns.  The  Antenna  on  the  left  represents  an  isotropic  Omni-directional  antenna,  in 
which  the  beam  radiates  equally  360°.  In  contrast,  the  antenna  pattern  on  right  represents 
a  highly  focused  beam  that  intensifies  power  in  a  particular  direction.  The  narrower  the 
beam  is,  the  higher  the  gain  is  (and  the  range),  because  you  eliminate  more  unwanted 
emissions  and  background  noise  in  the  other  directions. 


Figure  7:  Directional  versus  Omni-directional 
2.4.3,  Range  Factors 

The  receiver  ultimately  determines  the  performance  of  the  wireless  link.  The  link 
range  depends  on  the  sensitivity  of  the  receiver.  A  receiver’s  sensitivity  is  a  measure  of 
its  ability  to  discern  low-level  signals  and  still  correctly  translate  it  into  data.  A  signal 
cannot  be  processed  if  the  noise  magnitude  added  by  the  receiver  is  larger  than  that  of  the 
received  signal.  The  lower  the  sensitivity  level  of  the  receiver  the  better  the  hardware. 
For  example,  a  receiver  with  a  sensitivity  of  -100  dBm  is  better  than  a  receiver  sensitivity 
of  -93  dBm,  thus  able  to  hear  a  weaker  system.  Also,  receivers  require  a  minimum 


29 


Signal-to-Noise  (SNR)  ratio  to  successfully  decode  the  received  signal.  SNR  defines  the 
difference  of  power  in  the  receiver  between  a  meaningful  signal  and  background  noise. 
Note  dBm  is  an  abbreviation  for  the  power  ratio  in  decibels  (dB)  of  the  measured  power 
referenced  to  one  milliwatt. 

It  is  a  well-known  fact  that  power  is  a  precious  resource  in  MANETs.  Nodes  are 
usually  powered  by  batteries  that  are  constrained  by  weight  and  size.  High  transmit 
power  emission  will  likely  drain  the  battery  faster  or  cause  interference  between  nodes  in 
close  proximity.  In  fact,  energy  constraints  affect  almost  all  wireless  network  protocols  in 
some  manner,  so  energy  consumption  must  be  optimized  over  all  aspects  of  the  network 
design  [37]. 

Additionally,  the  radio  transmission  range  is  affected  by  the  environment  in  such 
a  complex  way,  which  makes  it  very  difficult  to  predict  the  behavior  of  the  network. 
Transmission  paths  and  parameters  change  instantaneously  due  to  obstructions  and 
environmental  changes.  Particularly,  radio  signal  propagation  is  subject  to  diffraction, 
reflection,  and  scattering.  In  the  case  of  diffraction,  the  signal  bends  around  an  object 
causing  sharp  irregularities  by  obstructing  the  path  between  transmitter  and  receiver. 
Reflection  occurs  when  signal  waves  strikes  the  earth  surface,  buildings,  and  walls  get 
reflected.  Finally,  scattering  is  caused  by  very  small  obstacles  such  as  rough  surface, 
foliage,  lampposts,  street  signs,  etc.  Hogie  et  al.  [10]  stated  no  simulator  implements  all 
three  properties  of  radio  propagation. 

Attenuation  is  another  important  factor  that  impacts  the  range  of  a  radio 
transmission.  It  is  the  decrease  of  signal  strength  between  the  transmitter  and  the  receiver. 
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In  the  air,  the  attenuation  is  simply  proportional  to  the  square  of  the  distance.  So,  when 
the  distance  doubles,  the  signal  becomes  one-fourth  less  strong.  In  addition  to  distance, 
environment  conditions  make  the  attenuation  change  over  time. 

2.5.  Chapter  Summary 

This  chapter  discussed  the  background  material  relating  to  the  development  and 
execution  of  Mobile  Ad  Hoc  Network  (MANET)  simulations.  The  first  part  of  this 
literature  review  presented  infonnation  on  the  state  of  simulation  relating  to  the  MANET 
community,  and  provides  insight  into  the  best  practices  or  approaches  to  prepare  and 
develop  a  simulation  that  produces  credible  results.  The  second  part  of  this  literature 
review  provided  infonnation  on  the  development,  execution,  and  analysis  of  recent 
airborne  MANET  simulations. 
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Ill  Methodology 


3.1.  Introduction 

Chapter  3  defines  the  methodology  used  to  conduct  a  performance  evaluation  of  a 
UAV  simulation  model  that  emulates  an  ad  hoc  network  during  actual  test  flights.  The 
systematic  approach  of  the  performance  evaluation  used  in  this  paper  allows  the 
experiment  to  be  independently  repeated,  if  desired. 

The  main  research  objective  is  to  develop  a  verified  simulation  network  model 
that  emulates  UAV  behavior  during  flight  by  applying  wireless  simulation  best  practices 
and  lessons  learned  identified  by  the  Mobile  Ad  Hoc  Network  (MANET)  simulation 
community.  Properly  validating  a  simulation  against  the  real-world  implementation  and 
environment  can  mitigate  many  of  the  problems  associated  with  simulation  modeling, 
such  as  incorrect  parameter  settings  and  improper  level  of  detail  [8], 

3.2.  Problem  Definition 

3.2.1.  Goals  and  Hypothesis 

There  are  no  well-accepted  procedures  to  validate  wireless  propagation  models 
and  user  mobility  models  in  MANET  simulations.  The  goal  of  this  research  is  to 
accurately  characterize  the  behavior  of  an  UAV  communication  network  by  incorporating 
the  effects  of  antenna  radiation  pattern,  signal  interference,  and  flight  mobility,  as  well  as 
developing  a  validated  simulation  model.  The  continuous  movement  of  a  UAV  makes  it 
challenging  to  accurately  duplicate  this  highly  complex  and  dynamic  communication 
network  in  a  simulation  model;  we  must  take  into  account  six  degrees  of  freedom: 
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latitude,  longitude,  altitude,  roll,  pitch,  and  yaw.  To  realistically  replicate  a  network  in 
simulation  all  six  degrees  of  freedom  of  an  airborne  mobile  node  are  modeled.  This  study 
also  identifies  the  effects  of  inherent  limitations  of  a  UAV  network  that  may  impact 
mission  effectiveness. 

Historically,  MANET  researchers  largely  focused  on  analyzing  the  effect  of  only 
three  degrees  of  movement  for  a  mobility  node  (latitude,  longitude,  and  altitude).  This 
research  extends  the  analysis  to  investigate  the  impact  of  three  additional  factors:  roll, 
pitch,  and  yaw.  By  accurately  incorporating  the  above  factors  in  this  research,  the 
underlining  physical  settings  that  have  a  significant  impact  of  the  accurately  of  UAV 
simulation  model  are  recognized,  and  a  realistic  representation  of  the  network  is  captured. 

This  study  determines  the  simulation  parameters  that  reflect  the  behavior  of  a 
UAV  network’s  physical  layer  by  adhering  to  strict  validation  procedures.  Thus, 
demonstrating  that  using  strict  validation  techniques  a  credible  simulation  model  can  be 
achieved. 

3.2.2.  Approach 

The  performance  metrics  collected  from  the  network  characterization  herein  are 
compared  to  existing  UAV  flight  test  data.  These  comparisons  provide  insight  into  the 
factors  and  parameters  that  affects  the  validity  of  a  UAV  simulation.  The  primary 
objective  is  to  develop  a  realistic  and  credible  simulation  model.  By  evaluating  the 
sensitivity  of  simulation  parameters  in  OPNET,  understanding  how  varying 
configurations  change  the  behavior  and  perfonnance  of  an  operational  UAV  network  in 
various  scenarios  will  be  gained.  Therefore,  extensive  validation  is  accomplished  and 
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parameters  critical  to  the  accuracy  of  this  simulation  characterization  are  added  as 
needed.  Thus,  an  iterative  approach  is  used  to  produce  the  final  model  and  appropriate 
settings.  Data  collected  from  actual  flight  tests  is  used  to  design  the  simulation,  and 
detennine  whether  the  model  behaves  as  a  UAV  network  does. 

The  results  answer  the  question:  how  do  the  effects  of  node  mobility,  antenna 
occlusion,  and  interference  impact  an  airborne  network?  This  approach  determines 
whether  the  aforementioned  simulation  attributes  accurately  represent  real-world 
implementation  of  the  UAV  network  modeled  in  this  research. 

3.3.  System  Boundaries 

Figure  8  shows  the  system  under  test  for  this  research  is  a  UAV  network  that 
includes  a  mobile  aerial  node  and  a  communication  link  with  a  stationary  ground  station 
receiver.  This  study  focuses  on  evaluating  the  steady-state  behavior  of  the  UAV  network 
physical  layer.  The  UAV  sends  data  to  the  ground  station  using  a  wireless  radio  adaptor. 
Performance  is  measured  by  various  characteristics  of  the  data  packets  successfully 
received  by  the  ground  station.  The  impact  of  node  mobility  and  the  antenna  radiation 
pattern  on  link  quality  during  several  points  throughout  a  flight  is  determined. 

The  component  under  test  (CUT)  is  the  physical  layer  of  the  network.  To  limit  the 
scope  of  this  research,  the  upper  OSI  layers  (Layers  2-7)  are  modeled  as  a  simple 
constant  rate  source  sending  a  packet  stream.  Figures  9  and  10  shows  the  transmitter  and 
receiver  physical  layer  node  models  implemented  in  OPNET. 
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Figure  8:  The  UAV  Network 


Figure  9:  Transmitter  Node  Model 

Isolating  the  evaluation  to  the  physical  layer  allows  a  comprehensive  analysis  of 
the  fundamental  characteristics  that  make  ad  hoc  mobile  networks  significantly  different 
from  traditional  wired  networks. 
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Figure  10:  Receiver  Node  Model 


3.4.  System  Services 

The  system  offers  one  service,  data  transmission  from  a  single  UAV  to  a 
stationary  ground  station.  The  UAV  network  transmits  data  to  a  ground  station  receiver. 
There  are  three  possible  outcomes:  successful  delivery,  delivery  with  bad  blocks,  or 
delivery  failure.  Success  occurs  when  the  ground  station  successfully  receives  all  the 
transmitted  packets.  Delivery  with  bad  blocks  occurs  when  the  ground  stations  receives 
blocks  with  errors.  The  final  outcome,  delivery  failure,  occurs  when  the  ground  station 
does  not  receive  packet  transmission  from  the  UAV  platform.  All  three  outcomes  will 
occur  in  a  realistic  MANET.  These  results  determine  what  physical  layer  parameters 
settings  best  suit  an  airborne  UAV  network. 
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3.5.  System  Workload 

The  workload  for  this  system  is  the  data  packet  transmitted  from  the  UAV 
platform  over  the  communication  link  to  the  ground  station.  The  packet  payload  is  flight 
data  consisting  of:  local  time,  data,  GPS  fix  quality,  latitude,  longitude,  elevation,  speed, 
heading,  signal  strength,  roll,  pitch,  and  yaw.  This  workload  is  essential  to  this  study 
because  it  is  the  only  measurable  input  into  the  system.  For  purpose  of  this  study,  the 
packet  size  and  transmission  rate  remains  constant  to  limit  the  scope  in  order  to  focus 
specifically  on  the  physical  layer,  and  to  match  the  flight  test  data  collected  from 
previous  UAV  flight  tests.  The  performance  of  multi-hop  routing  protocols  is  not  of 
current  interest,  but  may  be  feasible  for  future  work  once  a  validated  UAV  simulation  is 
available.  The  simulation  reads  actual  flight  position  data  to  build  its  flight  path  used  in 
place  of  unrealistic  mobility  models,  such  as  the  RWM. 

3.6.  Performance  Metrics 

According  to  Heidemann  [9],  to  properly  validate  a  simulation  model  one  must 
accurately  define  performance  metrics  to  compare  simulation  model  results.  Adequately 
defining  metrics  ensures  that  simulation  model  performance  reflects  the  behavior  of  a 
realistic  environment  or  operation.  The  following  perfonnance  metrics  are  of  interest  in 
this  research: 

Throughput  -  is  the  rate  at  which  packets  are  sent  through  the  channel  in 
seconds.  It  is  represented  as  the  average  number  of  packets,  or  bits,  successfully  received 
by  the  receiver  per  second.  Throughput  is  calculated  by  OPNET  using 
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Throughput  =  Packet_Average/Sim_Time  (3) 

where  Packets _Aver age  is  the  number  of  packets  accepted  to  date,  while  Sim  Time  is  the 
current  cumulative  simulation  time. 

Received  Power  -  is  a  key  factor  in  determining  if  the  receiver  correctly  captured 
the  packet  information.  Received  power  is  only  calculated  for  packets  that  are  classified 
as  valid.  This  metric  represents  the  average  power  of  a  packet  arriving  at  a  receiver 
channel.  The  received  power  is  updated  at  the  start  of  each  packet,  and  drops  to  zero 
when  the  packet  ends. 

Signal-to-Noise  Ratio  (SNR)  -  is  an  indication  of  potential  background 
interferences  with  the  wireless  signal.  It  is  typically  calculated  as  the  ratio  of  a  signal 
power  to  in-band  noise  power,  measured  at  the  receiver  location.  The  higher  the  ratio,  the 
less  interference  the  background  noise  causes.  A  research  goal  is  to  measure  the 
propagation  delay  experienced  by  the  network  as  a  function  of  the  SNR  (dB)  to  show  the 
expected  trend  in  delay  performance  at  the  physical  layer.  SNR  is  calculated  in  OPNET 
by  using 

SNR  =  10.0  *  log  10  (rcvd_power  /  (accum_noise  +  bkg_noise)))  (4) 

where  rcvd _power  is  the  average  received  power;  accum  noise  is  the  accumulated 
average  power  of  all  interference  noise;  and  bkgjioise  is  the  average  power  of  all  the 
background  noise. 

Received  Signal  Strength  -  is  a  function  of  distance  from  the  transmitter  and  the 
signal  power  (dB)  to  the  receiving  antenna.  Signal  strength  decreases  with  distance  and  is 
related  to  SNR. 


38 


Error  Rate  -  is  defined  as  the  number  of  erroneous  bits  received  divided  by  the 
total  number  of  bits  received.  In  a  wireless  environment  there  are  many  factors,  such  as 
interference,  distance,  and  signal-to-noise  ratio,  which  contribute  to  a  high  bit  error  rate. 

Dropped  Packets  -  is  the  number  of  transmitted  packets  that  are  not  received  by 
the  destination  node.  This  metric  is  the  total  number  of  packets  transmitted  subtracted  by 
the  total  number  of  packets  successfully  received  by  the  destination  node. 


3.7.  Workload  Parameters 

Table  3  lists  the  workload  parameters  for  this  research.  The  primary  workload  is 
packet  transmission  from  a  UAV  transmitter  to  a  ground-station  receiver.  A  normal 
distribution  with  a  mean  of  250  bytes  (2000  bits)  is  used  to  provide  the  packet  size  within 
OPNET.  Packets  are  generated  using  the  “simple  source”  process  model.  The  packet 
interarrival  time  is  approximately  0.2  seconds.  These  parameters  were  chosen  based  on 
data  available  from  the  real-world  flights. 


Table  3:  Workload  Parameters 


PARAMETERS 

DESCRIPTION 

Packet  Size 

Packet  size  for  this  experiment  varies  to  emulate  realistic  data 
transmitted  by  the  real-world  test-bed.  The  payload  consists  of  such 
parameters  as:  latitude,  longitude,  direction,  elevations,  speed  reading, 
and  received  signal  strength  indication.  These  parameters  support  the 
development  objective  to  use  empirical  data  in  the  simulation  model 
design  process. 

Transmission  Rate 

This  is  the  rate  that  data  is  transmitted  over  the  communication  link. 

Data  is  transmitted  at  a  constant  rate  based  the  trace  model  scenario. 

This  parameter  is  instrumental  to  correctly  calculate  the  network 
throughput. 
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3.8.  System  Parameters 


The  system  parameters  are  those  characteristics  of  the  system  that  are  of  interest 
to  accomplishing  the  research  objectives.  Other  system  parameters,  such  as  transmit 
power,  protocol,  and  signal  type  will  be  configured  based  on  flight  test  data.  Table  4 
explains  the  system  parameters  used  in  this  research. 


Table  4:  System  Parameters 


SYSTEM  PARAMETER 

DESCRIPTION 

Antenna  Gain 

It  is  well-known  that  antenna  types  (e.g.,  Omni  or 
directional),  as  well  as  their  orientations  can  greatly  affect 
the  perfonnance  of  wireless  links  [38,  39].  A  realistic 
antenna  pattern  is  necessary  to  characterize  the  UAV 
network  behavior. 

Interference  Noise 

Interference  is  a  fundamental  aspect  of  wireless  networks. 
Understanding  interference  effects  is  essential  to  modeling 
the  wireless  network  performance. 

Mobility  Trace 

The  trace  model  is  duplicated  from  real-world  flight  data. 

For  this  experiment  the  node  altitude  will  not  exceed  200 
meters.  Its  range  is  calculated  as  the  distance  of  the  UAV 
platform  from  the  ground  station.  The  UAV  range  does  not 
exceed  line  of  sight.  The  UAV’s  speed  varies  based  on  the 
scenarios  type  being  modeled.  Speeds  for  this  experiment 
do  not  exceed  50  knots. 

Historically,  MANET  research  studies  made  unrealistic 
assumptions  regarding  node  mobility  [10].  A  realistic 
mobility  model  is  necessary  to  accurately  characterize  the 
network  behavior.  If  possible  use  trace  models  or  realistic 
data  to  represent  the  mobile  node  behavior. 

Propagation  Model 

RF  Propagation  is  another  importance  aspect  to  conducing 
creditable  MANET  simulation  research.  Radio  Propagation 
is  more  difficult  to  model  than  wired  channels.  This 
research  model  considers  diffraction,  refraction,  and 
scattering  effects  on  transmission  quality. 

#  of  UAVs 

#  of  UAVs  is  the  number  of  mobile  nodes  transmitting 
packets  to  the  fixed  ground  station. 
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3.9.  Factors 


The  factors  are  a  subset  of  the  parameters  that  are  varied  during  experimentation. 
Table  5  outlines  the  factors  for  this  research.  These  factors  support  the  primary  focus  of 
our  research,  which  is  to  analyze  the  impact  of  node  mobility  and  antenna 
characterization  on  wireless  network.  Our  experiment  is  organized  into  one  main  scenario 
that  consists  of  three  separate  antenna  configurations  that  is  evaluated  against  three  trace 
models. 


Table  5:  System  Factors 


SYSTEM 

PARAMETER 

DESCRIPTION 

Antenna  Radiation 
Pattern 

Antenna  radiation  pattern  is  a  3-D  plot  of  the  relative  field  strength 
transmitted  from  or  received  by  the  antenna. 

Trace  Model 

Adjust  the  simulation  mobility  model  settings:  roll,  pitch,  and  yaw. 

This  experiment  uses  three  different  flight  scenarios  based  on  actual 
flight  data  to  evaluate  and  validate  system  behavior. 

3.10.  Evaluation  Technique 

This  study  compares  simulation  results  with  actual  UAV  network  performance 
results.  It  verifies  that  the  simulation  model  is  realistic,  by  ensuring  it  exhibits  the 
operational  UAV  flight  behavior.  In  Chapter  4  we  provide  a  comparison  of  aggregate 
statistical  measurements  from  actual  flights  tests  and  the  simulation  runs  to  provide  a 
useful  picture  to  make  our  assessment. 

OPNET  modeler  14.0  [25]  characterizes  the  system  under  test  behavior.  OPNET 
is  a  simulation  tool  that  includes  hundreds  of  pre -built  models  to  study  the  performance 
of  communication  networks.  The  tool  has  an  extensive  wireless  capability,  which 
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provides  the  ability  to  model  all  wireless  transmission  aspects,  including  radio  frequency 
propagation,  interference,  transmitter/receiver  characteristics,  and  node  mobility. 

OPNET  allows  the  system  under  test  to  be  evaluated  by  simulation  in  lieu  of  more 
costly  alternatives,  such  as  using  a  large  scale  test-bed  or  an  operational  system  to 
conduct  the  experiment.  OPNET  offers  many  standard  communication  components  that 
are  useful  and  provide  significant  efficiencies  when  constructing  complex 
communication  models.  However,  it  is  important  for  users  to  be  familiar  with  sub-model 
limitations  in  the  OPNET  standard  library,  such  as  using  model  components  developed  or 
validated  for  a  given  scenario. 


3.11.  Experimental  Design 

The  scenario  used  in  this  study  consists  of  a  single  UAV  transmitting  data 
packages  to  a  single  ground  station,  both  the  transmitter  and  receiver  are  equipped  with  a 
Microhard  320  Ultra  High  Frequency  hopping  wireless  modem.  Table  6  lists  the  modem 
specifications  important  to  this  study. 


Table  6:  Microhard  Specifications 


PARAMETERS 

VALUE 

Frequency 

310-390  MHz 

Channel  Bandwidth 

Depends  on  link  rate 

Selectable  Channels 

16,000  at  6.25kHz 

Range 

60+  miles  (dependent  on  link  rate  and  line 
of  sight) 

Sensitivity 

-107dBm  @  1 15.2kbps  link  rate 
-1 15dBm  @  19.2kbps  link  rate 

Output  Power 

lOOmW  -  1W 
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The  simulated  UAV  flies  in  a  flight  based  on  three  trace  models  recorded  from 
real-world  test  flights.  The  UAV  GPS  positional  data  and  the  static  ground  node 
coordinates  allow  the  analysis  of  various  performance  parameters  as  functions  of 
distance.  The  UAV  generates  a  normally  distributed  stream  of  250-byte  packets.  The 
purpose  for  having  a  single  node  UAV  as  the  sole  transmitter  is  to  avoid  possible  packet 
collision  due  to  simultaneous  transmissions.  The  ground  node  captures  the  broadcast 
packets  and  records  it  transmit  timestamp,  sequence  number,  size,  and  receive  signal 
strength  indication  (RSSI).  A  typical  scenario  lasts  from  12-15  minutes.  The  UAV  is 
controlled  by  autopilot  with  a  predetennined  flight  path.  The  simulated  UAV  node  moves 
according  to  a  predetermined  flight  path.  A  trajectory  file  is  loaded  into  OPNET  to  pennit 
the  node  to  move  according  to  the  latitude,  longitude,  and  altitude  of  real-world  test 
flights.  Table  7  lists  the  experiment  parameters  used  in  the  simulation  model  that  allow 
the  transmitter  and  receiver  to  establish  a  transmission  link;  according  to  the  real-world 
system  parameters. 


Table  7:  Fixed  System  Parameters 


PARAMETERS 

FIXED  VALUE 

Size  of  region 

1500m  x  1500m 

Path  loss  model 

default 

Data  rate  (bps) 

64,000 

Packet  formats 

Unfonnatted 

Bandwidth  (KHz) 

172 

Minimum  frequency  (MHz) 

370 

Modulation 

Binary  phase-shift  keying 

Receiver  processing  gain  (dB) 

-40  dB 

Transmitter  power  (W) 

1.0 

Noise  figure 

1.0 

Transmission  rate 

1  packet/.2  sec 
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Designing  a  realistic  antenna  model  is  also  essential  to  creating  a  credible 
simulation  model  that  behaves  as  the  real  system.  Vlah  [40]  conducted  a  study  on  antenna 
selection  perfonnance  in  802.1 1  networks  that  explored  the  use  of  antenna  selection  as  a 
new  avenue  of  perfonnance  improvement  in  mobile  wireless  ad  hoc  networks.  They  were 
able  to  show  that  antenna  selection  led  to  improved  perfonnance.  Table  8  lists  the 
important  antenna  attributes  used  in  our  study. 


Table  8:  Antenna  Attributes 


NAME 

VALUE 

Antenna  model 

Omni-directional 

Receiver  Antenna  Gain 

-.3dBm 

Transmitter  Antenna  Gain 

+3  dBm 

Ground  Antenna  pointing  ref.  theta 

180  degrees 

Ground  Station  altitude 

396  meters 

Ground  Station  latitude 

41.885 

Target  longitude 

-71.944 

Transmission  power 

1  Watt 

OPNET  Antenna  Editor  was  used  to  create  the  antenna  models  used  in  this 
research,  based  off  of  the  actual  test-bed  antennas.  The  ground  station  consists  of  an 
Omni-directional  antenna  with  an  average  gain  of  -.3dB.  The  ground  station  site  has  a  -40 
dB  attenuation  due  to  cable  loss.  The  UAV  consists  of  an  Omni-directional  antenna  with 
an  average  gain  of  +3dB. 

We  utilize  three  antenna  configurations  under  test  in  this  study.  Antenna  Setup  #1 
consists  of  a  detailed  antenna  pattern  for  both  the  UAV  and  ground  station.  While,  Setup 
#2  consists  of  a  detailed  antenna  pattern  for  the  UAV  node  and  less  detail  for  the  ground 
station.  Finally,  Setup  #3  consists  of  a  generic  isotropic  antenna  model  at  each  node, 
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allowing  for  a  360  degree  spherical  footprint.  Figures  11,  12,  and  13  display  the  custom 
antenna  radiation  patterns  used  in  this  study. 


UAV  Ground  Station 

Figure  11:  Antenna  Configuration  #1 


UAV  Ground  Station 

Figure  12:  Antenna  Configuration  #2 


UAV  Ground  Station 

Figure  13:  Antenna  Configuration  #3 
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A  full-factorial  design  in  three  levels  with  two  factors  is  used  for  this  experiment. 
It  is  a  common  experiment  design  in  which  every  setting  of  every  factor  appears  with 
every  setting  of  every  other  factor.  Also,  experiment  replication  provides  information  on 
variability.  Table  9  provides  the  simulation  random  seeds  used  in  experiment 
replications. 


Table  9:  Simulation  Random  Seeds 


FLIGHT 

RANDOM  SEEDS 

1 

107,337,  601,787,  929 

2 

821,  1409,  8803,6703,7159 

3 

79043,  99431,  39097,  149,  30977 

Full  factorial  for  this  experiment  consists  of  45  experiments.  This  number  of 
experiments  provides  the  appropriate  amount  of  data  to  statistically  validate  research 
goals  and  objectives.  Table  10  provides  a  breakdown  of  the  number  of  experiments 
performed.  There  are  three  mobility  trace  files  that  contain  flight  statistics  such  as 
latitude,  longitude,  and  altitude.  Also,  there  are  three  antenna  models  to  evaluate  how  the 
level  of  detail  of  an  antenna  radiation  pattern  impact  simulation  model  results. 


Table  10:  Factorial  Breakdown 


FACTORS 

LEVEL 

NUMBER 

UAV  Mobility 

Flight  Path  #1,  Flight  Path 
#2,  Flight  Path  #3 

3 

Antenna  Models 

Setup  #1,  Setup  #2,  Isotropic 
Antenna 

3 

Number  of  Replications 

Random  Seeds 

5 

Total  Experiments 

3*3*5 
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3,12.  Methodology  Summary 

This  study  answers  the  primary  question:  how  to  accurately  characterize  the  UAV 
network  behavior  that  is  impacted  by  node  mobility,  RF  propagation,  antenna  gain,  and 
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transmission  range.  This  characterization  provides  a  basis  for  developing  and  validating  a 
realistic  UAV  simulation  model.  This  experiment  provides  ample  results  to  provide  an 
empirical  comparison,  following  a  systematic  approach  for  conducting  a  perfonnance 
analysis  study.  All  aspects  of  the  UAV  characterization  and  experimental  design  were 
presented,  along  with  simulation  model  verification  and  validation. 
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IV  Results  and  Discussion 


4.1.  Chapter  Overview 

This  chapter  presents  the  results  and  analysis  on  the  performance  metrics 
collected  from  the  UAV  network  simulation  characterization  developed  within  this  study. 
Section  4.2  describes  validation  of  the  simulation  model  presented  in  this  research. 
Section  4.3  discusses  the  general  observations  related  to  the  experiment  results.  Section 
4.4  explains  the  statistical  analysis  techniques  used  to  evaluate  simulation  perfonnance. 
Section  4.5  concludes  with  a  summary  of  the  analysis  and  results. 

4.2.  Experiment  Validation  and  Verification 

Experiment  performance  metrics  are  compared  to  results  from  a  real-world  UAV 
network  implementation.  The  direct  comparison  of  simulation  results  against  actual  test 
flights  ensures  that  the  simulation  model  provides  meaningful  answers  to  the  research 
question  at  hand,  as  a  result  expanding  the  confidence  in  this  study.  As  explained  in 
Chapter  3  on  pages  35  and  36,  each  node  consists  of  a  wireless  network  adapter  and 
Omni-directional  antenna  that  represents  the  physical  layer  link  connecting  the 
transmitter  and  receiver.  As  expected,  the  direction  in  which  the  transmitter  travels  and 
the  distance  from  the  transmitter  to  the  receiver  are  significant  factors  to  the 
communication  link  quality.  The  primary  means  to  investigate  this  relationship  is  through 
receive  power  measurements,  or  the  receive  signal  strength  indictor  (RSSI),  recorded  at 
the  destination  node.  RSSI  is  a  common  feature  built  in  radios,  cell  phones,  and  wireless 
network  adapters  to  measure  the  incoming  signal.  Generally,  the  higher  the  RSSI  is  the 
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stronger  the  signal.  In  many  cases  this  measurement  can  help  detennine  the  aligmnent  of 
a  receiving  device  for  the  best  possible  signal  reception.  Both  wireless  modems  used  in 
this  study  have  this  capability. 

Establishing  a  radio  link  in  OPNET  depends  on  factors  such  as  the  node  altitude 
for  the  antennas,  transmission  signal  power,  modulation,  and  frequency.  The  receiver 
power  stage  of  the  OPNET  radio  transceiver  pipeline  computes  the  received  power  of  the 
arriving  packet’s  signal  (in  watts).  This  attribute  is  computed  by  taking  into  account  the 
initial  transmitted  power,  the  path  loss,  and  receiver  and  transmitter  antenna  gains. 
Received  power  is  calculated  by  OPNET  using: 

Rcvd  _  Power  =  in  _band  _tx_  power  x  tx_ant_gain  x  path  Joss  x  rx  _ant  _gain  ^ \ 

where  in  bandjx _power  is  the  amount  of  in-band  transmitter  power,  tx_ant_gain  is  the 
transmitter  antenna  gain  which  is  calculated  by  examining  the  vector  between  the 
transmitter  and  receiver,  path  loss  is  computed  as  a  function  of  wavelength  and 
propagation  distance,  while  rx_ant_gain  is  the  receiver  antenna  gain  which  is  calculated 
using  the  same  technique  as  the  transmitter  antenna  gain  model,  except  that  the  receiver- 
related  attributes  are  accessed.  This  study  uses  the  receiver  power  metric  results  from  the 
simulation  runs  to  compare  against  the  RSSI  data  collected  from  actual  test  flights  as  the 
method  to  validate  the  accuracy  of  the  UAV  simulation  model  herein.  Output  statements 
from  certain  phases  of  the  OPNET  radio  transceiver  pipeline  are  used  to  support 
simulation  validation.  Parameters  recorded  from  these  statements  are  compared  against 
the  real-world  network  representation,  consequently,  providing  evidence  that  the 
simulation  model  is  statistically  equivalent. 
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The  transmitter  node  defined  in  this  simulation  model  follows  a  predefined 
trajectory  file.  The  trajectory  file  consists  of  traversal-time  values  and  a  set  of  six¬ 
dimensional  (latitude,  longitude,  altitude,  roll,  pitch,  and  yaw)  coordinates  that  define  the 
UAV’s  flight  path.  The  procedure  for  creating  flight  paths  for  this  study  consists  of 
inputting  flights  coordinates  into  an  ASCII  text  file  with  a  .trj  extension  and  assigning  the 
file  to  the  UAV  node  using  the  “trajectory”  attribute  in  OPNET.  Appendix  A  contains  the 
MATLAB  code  necessary  to  create  a  trajectory  file. 


Figure  14:  Flight  Path  Trajectory  for  Flight  #1 

Using  mobility  patterns  that  most  accurately  represents  real-world  system  is 

essential  to  developing  a  simulation  model  that  is  useful  when  implemented  [14].  Figure 
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14  illustrates  the  flight  path  for  Flight  #1  of  this  experiment.  Additional  flight  paths  are 
included  in  Appendix  D. 

Early  pilot  tests  to  evaluate  the  effects  of  roll,  pitch,  and  yaw  on  communication 
link  quality  revealed  that  these  flight  attributes  had  no  impact  on  simulation  results. 
Effect  is  potentially  due  to  our  use  of  Omni-directional  antennas  in  this  experiment.  We 
would  expect  to  see  these  attributes  have  more  of  an  impact,  if  directional  antennas  are 
used. 

4.3.  General  Observations 

To  accurately  characterize  a  radio  link  that  existed  during  real-world  test  flights  is 
a  very  ambitious  aim  due  to  the  various  aspects  outside  the  control  of  our  experiment 
setup.  Factors  such  as  interference,  noise,  terrain,  environment  conditions,  and  RSSI 
variations  leads  to  some  expected  variance  from  the  real-world  link  recorded. 

4.3.1.  RSSI  Observations 

Recently,  there  have  been  a  number  of  studies  [41,  42]  investigating  the  viability 
of  using  RSSI  data  as  a  means  of  deriving  localization  of  a  transmitting  node.  In  this 
study,  RSSI  data  is  used  in  conjunction  with  node  positional  data  to  tune  the  simulation 
model  to  resemble  the  real-world  network.  Figures  15,  16,  and  17  are  a  set  of  time  series 
charts  from  test_flight_#  1  that  show  the  visual  difference  between  the  RSSI  values  of  the 
simulation  model  using  different  antenna  configurations.  In  general,  the  time  series  charts 
show  that  the  antenna  pattern  has  a  significant  impact  on  the  recorded  RSSI  values. 
Selecting  the  correct  level  of  detail  for  an  antenna  pattern  must  be  well  thought  out  by  the 
network  researcher.  Figure  17  illustrates  how  too  little  detail  can  produce  misleading 


51 


results.  The  RSSI  values  in  Figure  17  are  approximately  20  dBm  stronger  than  the  actual 
test  flight.  In  contrast,  Figure  15  shows  that  the  simulation  model  was  not  able  to  “fully 
characterize”  the  actual  test  flight,  but  a  strong  correlation  between  the  actual  flight  and 
simulation  model  was  achieved.  It  was  observed  that  as  the  antenna  model  was  fine-tuned 
to  resemble  reality,  there  was  an  increase  in  the  occurrence  of  outliners  in  the  model.  This 
effect  was  caused  by  not  having  the  actual  transmitter  and  receiver  antenna  radiation 
patterns  available,  just  a  general  description  was  provided.  Unfortunately,  solving  this 
problem  requires  complicated  antenna  engineering,  which  is  outside  the  scope  of  this 
study.  Supporting  graphs  for  Flight  #2  and  #3  are  included  in  Appendix  C. 


Entire  Flight  Snapshot  in  time 

Figure  15:  Comparison  of  RSSI  values  for  Flight  #1  vs  Antenna  setup  #1 


Entire  Flight  Snapshot  in  time 

Figure  16:  Comparison  of  RSSI  values  for  Flight  #1  vs  Antenna  setup  #2 
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Figure  17:  Comparison  of  RSSI  values  for  Flight  #1  vs  Isotropic  Antenna 
4.3.2.  Throughput  Observations 

This  study  shows  that  increasing  the  level  of  fidelity  of  the  antenna  radiation 
pattern  also  enhances  the  accuracy  of  throughput  calculation  produced  by  our  network 
simulation  model.  Figures  18,  19,  and  20  show  the  throughput  perfonnance  comparison 
between  Flight  #1  and  the  three  antenna  configuration  used  in  this  study.  Overall, 
Antenna  setup  #1  performed  closer  to  the  actual  system.  As  shown  in  Figure  14,  too  little 
detail  yields  unusable  results  that  cannot  be  used  to  answer  the  research  question  at  hand. 
In  each  case,  the  simulation  results  shown  in  Figures  18,  19,  and  20  does  not  exhibit  the 
throughput  variability  comparable  to  those  generated  by  the  real-world  test  flight.  These 
results  indicate  actual  model  use  is  dependent  on  much  more  than  accurately  modeling 
mobility,  antenna  patterns,  and  signal  strength.  This  could  be  attributed  to  our  inability  to 
characterize  several  key  aspects  of  the  test  flight.  For  instance,  aspects  such  as  RSSI 
variability,  radio  variability,  and  environmental  factors  could  not  be  reproduced.  Efforts 
to  reproduce  these  aspects  are  outside  this  phase  of  our  research. 
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Scatterplot  of  Throughput  vs  Distance  ( Flight  #1,  Setup  #1) 
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Figure  18:  Scatterplot  of  Flight  #1  vs  Setup  #1 


Scatterplot  of  Throughput  vs  Distance  ( Flight  #1,  Setup  #2) 
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Figure  19:  Scatterplot  of  Flight  #1  vs  Setup  #2 
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Scatterplot  of  Throughtput  vs  Distance  ( Flight  #1, 1  sotropic  Antenna) 
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Figure  20:  Scatterplot  of  Flight  #1  vs  Isotropic  Antenna 


The  authors  in  [41]  pointed  out  the  limitations  to  using  commodity  hardware  to 
capture  RSSI  measurement.  They  encountered  the  following  limitations  that  they  were 
able  to  work  around  to  some  degree,  such  as:  non-linearities  in  RSSI  measurements; 
invalid  RSSI  values;  missing  RSSI  values  in  deep  fades;  and  a  lack  of  foreign  RF 
interference  characterization.  The  above  limitations  are  certainly  factors  that  may  impact 
the  results  attained  in  our  research.  They  [41]  also  stated  that  while  a  simulator  can  replay 
a  captured  channel  trace,  it  can  only  do  so  at  a  very  coarse  timescale  and  with  far  less 
fidelity  than  a  physical  layer  emulator.  An  unintended  observation  from  our  research  was 
the  inability  to  replicate  the  fine  fidelity  required  to  fully  replicate  the  physical  layer  in  a 
simulation  environment.  Therefore,  the  research  in  our  study  was  only  able  to  observe  a 
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similar  trend  or  performance  as  the  real-world  system,  and  not  to  fully  replicate  the 
system  under  evaluation. 

4.3.3.  Distance  Observations 

Wireless  communication  is  a  broadcast  technology  and  depends  on  dynamically 
changing  parameters,  such  as  distance  to  evaluate  the  possible  connectivity  between  a 
transmitter  and  receiver.  OPNET  use  the  distances  between  nodes  to  compute  link  effects 
such  as  propagation  delay,  interference,  and  received  power  levels.  The  transceiver 
pipeline  evaluates  the  possible  connectivity  between  the  receiver  and  transmitter  for  each 
packet  transmission.  The  position  of  the  nodes  is  a  significant  factor  in  a  establishing  a 
radio  link.  The  transceiver  pipeline  calculates  whether  the  transmitter  node  has  direct 
line-of-sight  to  the  receiver  node.  If  the  earth  surface  or  some  other  object  is  between  the 
two  nodes,  then  the  nodes  are  said  to  be  occluded  and  the  link  computation  is 
discontinued.  If  there  are  no  obstructions  between  the  nodes,  then  the  link  computation 
continues,  and  a  radio  link  is  possible.  OPNET  also  models  the  weakening  of  the  radio 
signal  as  it  propagates  from  the  source  site.  It  is  assumed  that  the  path  loss  is  directly 
related  to  the  reciprocal  of  the  distance  squared. 

Again,  it  is  immediately  obvious  from  Figures  21,  22,  and  23  that  the  simulation 
model’s  RSSI  values  do  not  demonstrate  the  same  variability  as  the  real-world  test  flight. 
However,  these  figures  illustrate  how  the  simulation  model  behaviors  in  a  similar  matter 
to  the  real-world  system.  Through  simulation  we  were  not  able  to  completely  replicate 
the  fine  fidelity  exhibited  by  the  actual  test  flight,  but  are  able  to  make  general  inference 
on  the  simulation  network  performance.  Figure  21  shows  the  relationship  between 
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antenna  setup  #1  and  the  actual  test  flight  again  performing  more  like  the  system  under 
evaluation,  in  comparison  to  the  other  antenna  configurations. 


Scatterplot  of  Distance  vs  RSSI  (Test_  Flight_  #1,  Setup_  #1) 
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Figure  21:  Scatterplot  of  Distance  vs  RSSI  (Flight  #1,  Setup  #1) 


Scatterplot  of  Distance  vs  RSSI  (Test_  Flight_  #1,  Setup_  #2) 
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Figure  22:  Scatterplot  of  Distance  vs  RSSI  (Flight  #1,  Setup  #2) 
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Scatterplot  of  Distance  vs  RSSI  (Test_  Flight_  #1, 1  sotropic_  Antenna) 


0  100  200  300  400  500  600  700  800  900 

Distance  (meters) 


Figure  23:  Scatterplot  of  Distance  vs  RSSI  (Flight  #1,  Isotropic  Antenna) 

4.4.  Data  Analysis 

Data  analysis  consists  of  results  from  45  individual  runs  evaluating  the 
performance  of  three  flight  paths  and  three  antenna  configurations.  The  independent  two- 
sample  t-test  is  used  to  make  inferences  about  the  difference  between  the  means  of  the 
real-world  test  flights  and  the  simulation  model.  The  two-sample  t-test  assesses  whether 
the  means  of  two  systems  are  statistically  different  from  each  other.  This  technique  uses 
the  null  hypothesis  that  the  difference  between  two  population  means  is  equal  to  a 
hypothesized  value  and  tests  it  against  an  alternative  hypothesis.  Confidence  intervals  for 
the  mean  of  differences  using  the  two-sample  t-test  are  calculated  using: 


Confidence  interval  =  X,  -  X2  ±  tDF  x 


(6) 
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where  Xi,  s  1 ,  and  ni  are  the  mean,  standard  deviation  and  number  of  observations  for 
population  1;  while  X2,  S2,  and  n2  are  the  mean,  standard  deviation  and  number  of 
observations  for  population  2;  t  is  the  t-distribution  with  degrees  of  freedom;  where  a/2  is 
calculated  as: 


n  /  (100 -%  confidence  interval) 
/2  9 


(V) 


and  the  degrees  of  freedom  (D.F.)  is  calculated  using: 


(si. 


+  2/  ) 


D.F. 


n. 


( %  )2/(ni-l)  +  (s^  )2/(n2  -  1) 


(8) 


n. 


Confidence  intervals  of  the  mean  differences  that  contain  zero  suggest  no 
difference  between  two  systems.  Whereas,  for  confidence  intervals  that  do  not  contain 
zero,  there  is  a  statistically  significant  difference  between  the  two  systems. 


4.4.1.  Flight  #1  Analysis 

The  perfonnance  metrics  under  investigation  are  RSSI  and  throughput.  Figure  24 
is  an  interval  plot  of  the  average  RSSI  versus  antenna  radiation  pattern  configuration. 
This  plot  shows  the  95%  confidence  interval  centered  about  the  mean  for  each  antenna 
configuration.  Table  11  lists  the  antenna  setup,  their  means,  and  the  95%  confidence 
interval,  complete  statistical  results  for  all  three  test  flights  are  included  in  Appendix  B. 
Because  there  is  no  overlap  of  confidence  intervals  between  Flight  #1,  Setup  #2,  and  the 
isotropic  antenna,  the  groups  are  statically  different.  Because  the  mean  values  for  Flight 
#1  and  Setup  #1  fall  within  the  confidence  intervals  of  each  other,  suggests  that  their 
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RSSI  values  are  not  significantly  different.  Recall  that  Flight  #1  is  the  real-world  flight 
and  Setup  #1  is  the  detailed  antenna  configuration. 


Figure  24:  Interval  Plot  RSSI  vs  Antenna  (Flight  #1) 


Setup  #2’s  mean  average  RSSI  value  (-96.14)  is  slightly  lower  than  those 
observed  by  Flight  #1  (-93.31).  Whereas  the  isotropic  antenna’s  mean  average  RSSI 
value  (-73.44)  is  much  higher.  Results  of  the  t-test  for  Flight  #1  and  Setup  #1  generates  a 
confidence  interval  of  (-0.208,  0.410)  which  includes  zero.  Since  this  confidence  interval 
contains  zero,  Flight  #1  and  Setup  #1  are  not  significantly  different.  The  p-value  (0.521) 
for  Setup  #1  is  greater  than  the  alpha  value  0.05,  which  also  confirms  that  there  is  no 
evidence  of  a  difference,  and  the  simulation  scenario  can  be  considered  statistically 
equivalent. 


Table  11:  Average  RSSI  vs  Antenna  Configuration  (95%  Confidence  Intervals) 


Antenna 

Mean  (RSSI) 

Confidence  Range 

p-value 

Statically 

equivalent 

1 

-93.41 

(-0.208,0.410) 

0.521 

Y 

2 

-96.14 

(2.555,3.120) 

0.000 

N 

Isotropic 

-73.44 

(-20.117,-19.607) 

0.000 

N 
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Figure  25:  Interval  Plot  of  Throughput  vs  Antenna  (Flight  #1) 


Figure  25  is  an  interval  plot  of  the  throughput  versus  the  antenna  radiation  pattern 
configuration.  This  plot  shows  the  95%  confidence  interval  centered  about  the  mean  for 
the  antenna  configuration.  Table  12  lists  the  antenna  setup,  their  means,  and  the  95% 
confidence  interval.  Since  there  is  no  overlap  of  confidence  intervals  between  Flight  #1, 
and  the  isotropic  antenna,  suggest  that  the  groups  are  statically  different.  Because  the 
mean  values  for  Flight  #1,  Setup  #1,  and  Setup  #2  fall  within  the  confidence  intervals  of 
each  other,  their  throughput  values  are  significantly  equivalent. 

The  isotropic  antenna  mean  throughput  (56,423)  is  much  higher  than  those 
observed  by  Flight  #1  (52,139),  as  well  as  the  other  groups.  This  result  is  caused  by  the 
simplistic  design  of  the  isotropic  antenna.  Results  of  the  t-test  for  Flight  #1  and  Setup  #1 
generates  a  confidence  interval  of  (-223,  1464).  The  t-test  for  Flight  #1  and  Setup  #2 
generates  a  confidence  interval  of  (-711,  868).  Because  these  confidence  intervals  contain 
zero,  Flight  #1,  Setup  #1,  and  Setup  #2  are  not  significantly  different.  The  p-values  for 
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both  Setup  #1  (0.149)  and  Setup  #2  (0.845)  are  greater  than  the  alpha  value  0.05,  which 
confirms  that  there  is  no  evidence  of  a  difference,  and  the  simulation  scenario  can  be 
considered  statistically  equivalent. 


Table  12:  Throughput  vs  Antenna  Configuration  (95%  < 

Confidence  Intervals) 

Antenna 

Mean  (bps) 

Confidence  Range 

p-value 

Statically 

equivalent 

1 

52,518 

(-223,  1464) 

0.149 

Y 

2 

52,090 

(-711,  868) 

0.845 

Y 

Isotropic 

56,423 

(-4980,  -3588) 

0.000 

N 

4.4.2.  Flight  #2  Analysis 

Figure  26  is  an  interval  plot  of  the  average  RSSI  versus  antenna  radiation  pattern 
configuration.  This  plot  shows  the  95%  confidence  interval  centered  about  the  mean  for 
each  antenna  configuration.  Table  13  lists  the  antenna  setup,  their  means,  and  the  95% 
confidence  interval.  Because  there  is  no  overlap  of  confidence  intervals  between  Flight 
#2,  Setup  #2,  and  the  isotropic  antenna,  the  groups  are  statistically  different.  The  mean 
values  for  Flight  #2  and  Setup  #1  fall  within  the  confidence  intervals  of  each  other, 
suggest  that  their  RSSI  values  are  significantly  equivalent.  Setup  #2’s  mean  average 
RSSI  value  (-96.76)  is  slightly  lower  than  those  observed  in  Flight  #2  (-93.91).  While  the 
isotropic  antenna’s  mean  average  RSSI  value  (-73.69)  is  much  higher.  Results  of  the  t- 
test  for  Flight  #2  and  Setup  #1  generates  a  confidence  interval  of  (-0.425,  0.219)  which 
includes  zero.  Since  this  confidence  interval  contains  zero,  Flight  #1  and  Setup  #1  are  not 
significantly  different.  The  p-value  (0.530)  for  Setup  #1  is  greater  than  the  alpha  value 
0.05,  which  also  confirms  that  there  is  no  evidence  of  a  difference,  and  the  simulation 


scenario  can  be  considered  statistically  equivalent. 
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I  nterval  Plot  of  Average  RSSI  vs  Antenna  Configuration 

95%  Cl  for  the  Mean 


Figure  26:  Interval  Plot  of  RSSI  vs  Antenna  (Flight  #2) 


Table  13:  Average  RSSI  vs  Antenna  Configuration  (95%  Confidence  Intervals) 


Antenna 

Mean  (RSSI) 

Confidence  Range 

p-value 

Statically 

equivalent 

1 

-93.41 

(-0.425,0.219) 

0.530 

Y 

2 

-96.76 

(2.543,3.153) 

0.000 

N 

Isotropic 

-73.69 

(-20.527,-19.917) 

0.000 

N 

Figure  27  is  an  interval  plot  of  the  throughput  versus  the  antenna  radiation  pattern 
configuration.  This  plot  shows  the  95%  confidence  interval  centered  about  the  mean  for 
the  antenna  configuration.  Table  14  lists  the  antenna  setup,  their  means,  and  the  95% 
confidence  interval.  Since  there  is  no  overlap  of  confidence  intervals  between  Flight  #2, 
Setup  #2,  and  the  isotropic  antenna,  the  groups  are  statically  different.  Because  the  mean 
values  for  Flight  #2,  and  Setup  #1  fall  within  the  confidence  intervals  of  each  other,  their 
throughput  values  are  not  significantly  different. 
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Figure  27:  Throughput  vs  Antenna  (Flight  #2) 


The  isotropic  antenna  mean  throughput  (42,385)  is  slightly  higher  than  those 
observed  by  Flight  #2  (40,030),  as  well  as  the  other  groups.  From  our  observation,  this 
result  is  caused  by  the  simplistic  design  of  the  isotropic  antenna.  Results  of  the  t-test  for 
Flight  #2  and  Setup  #2  generates  a  confidence  interval  of  (-2047,  -276),  thus  suggesting 
the  two  systems  are  significantly  different.  The  t-test  for  Flight  #2  and  Setup  #1  generates 
a  confidence  interval  of  (-144,  1732).  Since  this  confidence  intervals  contain  zero,  Flight 
#2  and  Setup  #1  are  not  significantly  different.  The  p-value  for  Setup  #1  (0.097)  is 
greater  than  the  alpha  value  0.05,  which  also  confirms  that  there  is  no  evidence  of  a 
difference,  and  the  simulation  scenario  can  be  considered  statistically  equivalent. 


Table  14:  Throughput  vs  Antenna  Configuration  (95% 

Confidence  Intervals) 

Antenna 

Mean  (bps) 

Confidence  Range 

p-value 

Statically 

equivalent 

1 

39,236 

(-144,  1732) 

0.097 

Y 

2 

41,191 

(-2047,  -276) 

0.010 

N 

Isotropic 

42,385 

(-3231,-1479) 

0.000 

N 
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4.4.3.  Flight  #3  Analysis 

Figure  28  is  an  interval  plot  of  the  average  RSSI  versus  antenna  radiation  pattern 
configuration.  This  plot  shows  the  95%  confidence  interval  centered  about  the  mean  for 
each  antenna  configuration.  Table  15  lists  the  antenna  setup,  their  means,  and  the  95% 
confidence  interval.  Since  there  is  no  overlap  of  confidence  intervals  between  Flight  #3, 
Setup  #1,  Setup  #2,  and  the  isotropic  antenna,  the  groups  are  statistically  different. 


Figure  28:  Interval  Plot  of  RSSI  vs  Antenna  (Flight  #3) 


Setup  #l’s  mean  average  value  (-90.95)  is  slightly  higher  than  those  observed  in 

Flight  #3.  Setup  #2  mean  average  RSSI  value  (-93.99)  is  slightly  lower  than  those 

observed  by  Flight  #3  (-92.02).  Whereas  the  isotropic  antenna’s  mean  average  RSSI 

value  (-71.09)  is  much  lower.  Results  of  the  t-test  for  Flight  #3  and  Setup  #1  generates  a 

confidence  interval  of  (-1.366,  -0.803),  thus  not  including  zero.  Whereas,  results  of  the  t- 

test  for  Flight  #3  and  Setup  #2  generates  a  confidence  interval  of  (1.727,  2.224),  thus  not 

including  zero.  Where,  results  of  the  t-test  for  Flight  #3  and  the  isotropic  antenna 

generates  a  confidence  interval  of  (-21.159,  -20.706),  thus  not  including  zero.  Since  none 
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of  these  confidence  intervals  contains  zero,  Flight  #3  is  significantly  different  from  all  the 
groups,  and  cannot  be  successfully  simulated  within  our  current  model. 


Table  15:  Average  RSSI  vs  Antenna  Configuration  (95%  Confidence  Intervals) 


Antenna 

Mean  (RSSI) 

Confidence  Range 

p-value 

Statically 

equivalent 

1 

-90.95 

(-1.366,  -0.803) 

0.000 

N 

2 

-93.99 

(1.727,2.224) 

0.000 

N 

Isotropic 

-71.09 

(-29.159,  -20.706) 

0.000 

N 

Figure  29  is  an  interval  plot  of  the  throughput  versus  the  antenna  radiation  pattern 
configuration.  This  plot  shows  the  95%  confidence  interval  centered  about  the  mean  for 
the  antenna  configuration.  Table  16  lists  the  antenna  setup,  their  means,  and  the  95% 
confidence  interval.  Since  there  is  no  overlap  of  confidence  intervals  between  Flight  #3, 
Setup  #2,  and  the  isotropic  antenna,  suggest  that  the  groups  are  statistically  different. 
Because  the  mean  values  for  Flight  #3,  and  Setup  #1  fall  within  the  confidence  intervals 
of  each  other,  their  throughput  values  are  significantly  equivalent. 


Figure  29:  Interval  Plot  Throughput  vs  Antenna  (Flight  #3) 
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The  isotropic  antenna  mean  throughput  (56,050)  is  much  higher  than  those 
observed  by  Flight  #3  (52,959),  as  well  as  the  other  groups.  Again,  this  result  is  caused 
by  the  simplistic  design  of  the  isotropic  antenna.  Results  of  the  t-test  for  Flight  #3  and 
Setup  #1  generates  a  confidence  interval  of  (-632,  791).  Since  this  confidence  interval 
contains  zero,  these  systems  are  not  significantly  different.  The  p-value  (0.826)  for  Setup 
#1  is  greater  than  the  alpha  value  0.05,  which  also  confirms  that  there  is  no  evidence  of  a 
difference,  and  the  simulation  scenario  can  be  considered  statistically  equivalent.  The  t- 
test  for  Flight  #3  and  Setup  #2  generates  a  confidence  interval  of  (-2739,  -1461).  Because 
this  confidence  interval  does  not  contain  zero,  Flight  #3,  and  Setup  #2  are  significantly 
different,  and  cannot  be  considered  statistically  equivalent. 


Table  16:  Throughput  vs  Antenna  Configuration  (95%  < 

Confidence  Intervals) 

Antenna 

Mean  (bps) 

Confidence  Range 

p-value 

Statically 

equivalent 

1 

52,880 

(-632,  791) 

0.826 

Y 

2 

55,059 

(-2739,-1461) 

0.000 

N 

Isotropic 

56,050 

(.3704,  -2477) 

0.000 

N 

4.4.4.  Data  Excluded  from  Analysis 

Due  to  the  real-world  test-bed  data  collection  method  performed  by  the  research 
sponsor,  the  following  simulation  performance  metrics  could  not  be  validated  during  this 
experimentation  phase:  error  rate,  SNR,  and  dropped  packets.  The  test-bed  is  limited  to 
recording  the  number  of  good  blocks  received  between  bad  blocks.  This  collection 
approach  is  not  comparable  to  the  approach  OPNET  uses  to  record  these  measurements. 
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4.5.  Summary  of  Analysis  and  Results 

The  major  goals  of  this  study  were  to  develop  a  simulation  network  model  by 
using  the  best  simulation  practices  while  providing  evidence  that  model  level  of  detail 
and  node  mobility  has  a  significant  impact  on  simulation  results.  Network  research 
should  put  considerable  thought  in  how  one  designs  the  simulation  model  physical  layer. 
Too  little  design  will  provide  misleading  results,  while  attempting  to  incorporate  too 
much  detail  can  be  too  time  consuming  and  costly.  Through  following  a  systemic 
approach  we  were  able  to  demonstrate  that  the  antenna  model  with  the  greatest  level  of 
detail  (Setup  #1)  strongly  correlates  to  real-world  test  flights  in  two  of  the  three  flight  test 
under  evaluation  in  this  study.  As  we  have  noted,  there  were  some  instances  in  the 
network  simulation  that  did  not  match  the  test-bed  flights.  Preston  et  al.  [23]  encountered 
similar  results  in  their  enhanced  OPNET  models.  They  recommended  addressing  these 
inconsistencies  by  incorporating  an  accurate  antenna  pattern  from  the  actual  flying 
platform.  Recall,  a  general  antenna  description  was  only  available  at  the  time  of  this 
study. 

This  chapter  covered  the  validation  of  the  network  simulation  model  in  this  study, 
as  well  as  the  data  observation  and  analysis  to  explain  how  network  simulation  result 
relates  to  reality.  The  two-sample  t-test  was  used  to  statistically  validate  our  network 
simulation  model’s  accuracy. 
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V  Conclusions  and  Recommendations 


5.1.  Chapter  Overview 

This  chapter  concludes  the  documentation  of  the  research  performed.  Section  5.2 
presents  a  restatement  of  research  goals.  Section  5.3  presents  a  summary  of  the 
conclusions  drawn  from  the  analysis  and  results.  Section  5.4  discusses  the  significance  of 
this  research.  Section  5.5  discusses  recommendations  for  further  research,  and  Section 
5.6  briefly  summarizes  this  chapter. 

5.2.  Research  Goal 

The  goal  of  this  research  was  to  accurately  characterize  the  behavior  of  UAV 
communication  networks  by  incorporating  the  effects  of  antenna  radiation  patterns,  signal 
interference,  and  flight  mobility,  as  well  as  developing  a  validated  simulation  model.  The 
underlining  physical  settings  that  have  a  significant  impact  on  UAV  simulation  model 
accuracy  were  identified,  and  a  realistic  representation  of  the  network  was  captured. 
Consequently,  demonstrating  that  through  strict  validation  techniques,  a  credible 
simulation  model  can  be  achieved. 

5.3.  Conclusion  of  Research 

Several  conclusions  can  be  drawn  from  the  results  and  analysis  of  this  research. 
Through  sensitivity  analysis  of  the  essential  OPNET  radio  transceiver  attributes  and 
development  of  custom  antenna  models,  we  were  able  to  create  a  realistic  network 
simulation  model  to  provide  a  means  of  evaluating  the  airborne  network  physical  layer 
link.  By  incorporating  actual  flight  data  into  our  OPNET  model,  we  were  able  to  compare 

our  captured  data  results  against  actual  flight  observations  to  verify  the  accuracy  of  our 
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model.  In  all  cases  the  antenna  setup  with  the  least  level  of  detail  produced  behavior  not 
representative  of  the  real-world  system,  thus  providing  misleading  simulation  results. 
Both  RSSI  and  throughput  results  observed  from  the  isotropic  antenna  setup  from  all 
flights  could  mislead  researchers  into  believing  that  the  UAV  communication  link  quality 
is  better  than  it  is  actually  performing.  In  contrast,  the  perfonnance  of  antenna  setup  #1 
strongly  correlated  to  the  real-world  test  flight  in  two  out  of  the  three  test  flights 
investigated  in  this  study.  These  results  demonstrated  that  the  appropriate  level  of  detail 
must  be  put  into  designing  the  simulation  network  physical  level  to  avoid  misleading 
simulation  results. 

5.4.  Research  Significance 

This  study  developed  a  verified  network  model  that  emulates  UAV  network 
behavior  during  flight,  using  a  leading  simulation  tool.  A  flexible  modeling  and 
simulation  environment  was  developed  to  test  proposed  technologies  against  realistic 
mission  scenarios.  In  addition,  this  study  identified  the  essential  significant  parameters 
that  impact  the  simulation  network  model  physical  layer.  We  clearly  demonstrated  there 
is  an  interdependent  relationship  between  the  UAV  transmission  power  and  distance, 
channel  bandwidth,  and  antenna  radiation  pattern.  These  parameters  most  impacted  our 
simulation  performance.  Additionally,  this  research  contributes  a  validated  simulation 
model  to  the  MANET  community. 
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5.5.  Recommendations  for  Future  Research 


Further  research  is  warranted  into  the  use  of  receive  strength  signal  indicator 
using  a  wireless  model  that  has  been  calibrated  and  evaluated  to  identify  variability 
caused  by  hardware  and  environment  factors.  In  addition,  including  verified  antenna 
radiation  patterns  and  various  network  protocols  would  improve  the  level  of  confidence 
of  this  model. 

5.6.  Chapter  Summary 

This  chapter  presents  the  overall  conclusions  that  are  drawn  from  the  results. 
Research  significance  and  goals  were  discussed;  and  several  recommendations  for  the 
future  are  presented. 
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Appendix  A.  MATLAB  Trajectory  Source  Code 


%OPNET  Trajectory  File 

%This  file  write  raw  flight  data  into  an  OPNET  Trajectory  File 
%Created  on:  10/09/2008 

clc;  clear; 

fid  =  fopen ( ' Flight#l . txt ' )  ; 

%Read  raw  flight  data  into  text  file 
%UAS  data  columns 

%1  etime--  elapsed  time (sec) from  UTC  time  3 

%2  qGPS--  quality  of  PS  time  from  4 

%3  lat--  latitude  (deg)  from  5-6 

%4  long-  longitude  (deg) 7-8 

%5  elev —  elevation (m) 9 

%6  speed-- (kph) 10 

%7  head--  heading  (deg) 11 

%8  nsat--number  of  satellites  seen  by  GPS  12 

%9  roll--  UAV  roll (radians) 13 

%10  pitch--  UAV  pitch  (radians) 14 

%11  Uhead--  UAV  heading  (radians) 15 

%12  Utrack--  UAV  track  (radians) 16 

%13  RSS I  — (dBm) 17 

%14  pGood--  number  of  good  packets  since  last  data  point  18 
%15  pBad--  number  of  bad  packets  since  last  data  point  19 
%16  nByte--  number  of  bytes  received  since  last  data  point  20 


Data=textscan (fid. 


'9-Q9-Q9-Q9--f9--f9-^9--f9-^9--F9--f9--f9--f9--f9--F9--f9--F9--F9--F9--f9--F9-QS--FS--F 

oo  oo  oo  ol  ol  Ob  ol  Ob  ol  ol  ol  ol  ol  ol  ol  ol  ol  ol  ol  ol  oo  ol  ol 


9'-F9'-F9'-F9'-F9'-F9'-F9'-F9'-F  J 
ol  ol  ol  ol  ol  ol  ol.  ol 


' delimiter ' , 


fclose (fid) ; 

fid  =  fopen  ( ' Output . txt  ,  'wt'); 


%%%  Write  to 

fprintf (fid, 
fprintf (fid, 
fprintf (fid, 
fprintf (fid, 
fprintf (fid, 
fprintf (fid, 
fprintf (fid, 
fprintf (fid, 
fprintf (fid. 


.  trj  file  Headers 
' Version :  3\n ' ) ; 

'Position  Unit:  DegreesVn'); 

'Altitude  Unit:  meters\n'); 

'Coordinate  Method:  fixed\n'); 

' Altitude_Method :  absolute\n ' ) ; 

' locale :  C\n ' ) ; 

' Calendar_Start :  unused\n'); 

' Coordinate_Count :  %u\n',size(Data{l,l},l)); 

'#  X  Position  ,Y  Position  , Altitude 


fprintf (fid, ' 


, Traverse  Time 


,Wait  Time 


, Pitch 


fprintf (fid, '  , Yaw 


,  Roll 


\n ' 


%Write  flight  data  into  trajectory  file 
lat  =  Data{l, 5}/ (60*100000) ; 
lat_direction  =  Data {1,6}; 
long  =  Data{ 1, 7 }/ (60*100000) ; 
long_direction  =  Data {1,8}; 

Alt  =  Data { 1 , 9 } / 1 0 ; 
roll  =  Data { 1 , 13 } /1000 ; 
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pitch  =  Data { 1 , 14 } /1000 ; 
yaw  =  Data { 1 , 15 } /1000 ; 

RSSI  =  Data{ 1,17}; 
good_packets  =  Data {1,18}; 
bad_packets  =  Data{l,19}; 
bytes_received  =  Data {1,20}; 
total  =  size (Data{ 1, 1 } , 1)  ; 
for  n  =  1: total; 

if  strcmp('N',  lat  direction (n) ) 
lat(n)  =  1  *  lat(n); 

else 

lat(n)  =  -1  *lat(n); 

end 

if  strcmp('E',  long  direction (n) ) 
long(n)  =  1  *  long(n); 

else 

long(n)  =  -1  *long(n); 

end 

end 


for  count  =  1 :  total; 
fprintf (fid, '%.9f 
fprintf (fid, ' 
fprintf (fid, ' 
fprintf (fid, ' 
roll (count) )  ; 
end; 

fclose  (fid) ; 


,%.9f',  long(count),  lat (count)); 

,%.2f  , OhOml.Os  ,  Alt (count)); 

,0h0m0.0s  ,%f',  pitch (count) ) ; 

,%.5f  ,%.5f\n',  yaw(count). 


fidl  =  fopen  ('packet  summary.txt’,  'wt'); 

fprintf ( fidl ,' RSSI  ,#  of  Good_packets  ,#  of  Bad_packets 

, Received_packets\n ' ) ; 

for  count  =  1: total; 

fprintf ( fidl ,',  %d  ,  %d 1  ,  RSSI (count) ,  good_packets(count)); 

fprintf ( fidl ,  '  , %d  ,%d\n', 

bad_packets (count) ,  bytes_received (count) ) ; 
end; 

fclose  (fidl ) ; 

%Program  End 


73 


Appendix  B.  Supporting  Data 


This  appendix  provides  data  to  support  the  analysis  of  this  study.  The  two-sample 
t-test  results  for  all  flight  are  contained  in  this  appendix. 


B.l.  Flight  #1  Analysis 

Two-Sample  T-Test  and  Cl:  Flight  #1,  Setup  #1 

N  Mean  StDev  SE  Mean 
Flight  #1  3697  -93.31  5.52  0.091 

Custom_Antenna_l  3697  -93.41  7.84  0.13 

Difference  =  mu  (Flight  #1)  -  mu  (Setup  #1) 

Estimate  for  difference:  0.101 

95%  Cl  for  difference:  (-0.208,  0.410) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  0.64  P-Value  =  0.521  DF  =  6633 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Setup  #2 

N  Mean  StDev  SE  Mean 
Flight  #1  3697  -93.31  5.52  0.091 

Custom_Antenna_2  3697  -96.14  6.81  0.11 

Difference  =  mu  (Flight  #1)  -  mu  (Setup  #2) 

Estimate  for  difference:  2.837 

95%  Cl  for  difference:  (2.555,  3.120) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  19.68  P-Value  =  0.000  DF  = 
7086 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Isotropic  Antenna 

N  Mean  StDev  SE  Mean 
Flight  #1  3697  -93.31  5.52  0.091 

Isotropic  Antenna  3697  -73.44  5.67  0.093 

Difference  =  mu  (Flight  #1)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -19.862 

95%  Cl  for  difference:  (-20.117,  -19.607) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -152.63  P-Value  =  0.000  DF  = 
7386 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Setup  #1  (Snapshot) 


N  Mean 

StDev  SE 

Mean 

Flight  #1 

56  -77.64 

4.71 

0.63 

Setup  #1 

56  -77.51 

3.23 

0.43 

Difference  =  mu  (Flight  #1)  -  mu 
Estimate  for  difference:  -0.132 

(Setup  1) 

95%  Cl  for 

difference:  (-1.647, 

1.383) 
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T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -0.17  P-Value  =  0.863  DF  =  97 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Setup  #2  (Snapshot) 

N  Mean  StDev  SE  Mean 
Flight  #1  56  -77.64  4.71  0.63 

Setup_2  56  -82.09  3.23  0.43 

Difference  =  mu  (Flight  #1)  -  mu  (Setup  #2) 

Estimate  for  difference:  4.445 

95%  Cl  for  difference:  (2.930,  5.960) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  5.82  P-Value  =  0.000  DF  =  97 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Isotropic  Antenna  (Snapshot) 

N  Mean  StDev  SE  Mean 

Flight  #1  56  -77.64  4.71  0.63 

Isotropic  Antenna  56  -58.64  3.29  0.44 

Difference  =  mu  (Flight  #1)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -19.005 

95%  Cl  for  difference:  (-20.529,  -17.482) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -24.76  P-Value  =  0.000  DF  =  98 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Setup  #1  (Throughput) 

N  Mean  StDev  SE  Mean 
Flight  #1  3696  52139  20905  344 

Setup  #1  3696  51518  15710  258 

Difference  =  mu  (Flight  #1)  -  mu  (Setup  #1) 

Estimate  for  difference:  621 

95%  Cl  for  difference:  (-223,  1464) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  1.44  P-Value  =  0.149  DF  =  6859 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Setup  #2  (Throughput) 

N  Mean  StDev  SE  Mean 
Flight  #1  3696  52139  20905  344 

Setup  #2  3696  52060  12734  209 

Difference  =  mu  (Flight  #1)  -  mu  (Setup  #2) 

Estimate  for  difference:  79 

95%  Cl  for  difference:  (-711,  868) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  0.20  P-Value  =  0.845  DF  =  610 


Two-Sample  T-Test  and  Cl:  Flight  #1,  Isotropic  Antenna  (Throughput) 
N  Mean  StDev  SE  Mean 

Flight  #1  3696  52139  20905  344 

Isotropic  Antenna  3696  56423  5390  89 

Difference  =  mu  (Flight  #1)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -4284 
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0.000  DF 


95%  Cl  for  difference:  (-4980,  -3588) 

T-Test  of  difference  =  0  (vs  not  =)  :  T-Value  =  -12.06  P-Value  = 
4184 


B.2.  Flight  #2  Analysis 


Two-Sample  T-Test  and  Cl :  Flight  #2 ,  Setup  #1 

N  Mean  StDev  SE  Mean 
Flight  #2  3674  -93.91  7.20  0.12 

Setup  #1  3674  -93.81  6.87  0.11 

Difference  =  mu  (Flight  #2)  -  mu  (Setup  #1) 

Estimate  for  difference:  -0.103 

95%  Cl  for  difference:  (-0.425,  0.219) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -0.63  P-Value  =  0.530  DF  = 
7329 


Two-Sample  T-Test  and  Cl :  Flight  #2 ,  Setup  #2 

N  Mean  StDev  SE  Mean 
Flight  #2  3674  -93.91  7.20  0.12 

Setup  #2  3674  -96.76  6.08  0.10 

Difference  =  mu  (Flight  #2)  -  mu  (Setup  #2) 

Estimate  for  difference:  2.848 

95%  Cl  for  difference:  (2.543,  3.153) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  18.32  P-Value  =  0.000  DF  = 
7143 


Two-Sample  T-Test  and  Cl:  Flight  #2,  Isotropic  Antenna 

N  Mean  StDev  SE  Mean 

Flight  #2  3674  -93.91  7.20  0.12 

Isotropic  Antenna  3674  -73.69  6.10  0.10 

Difference  =  mu  (Flight  #2)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -20.222 

95%  Cl  for  difference:  (-20.527,  -19.917) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -129.87  P-Value  =  0.000  DF  = 
7150 


Two-Sample  T-Test  and  Cl:  Flight  #2,  Setup  #1  (Snapshot) 


N 

Mean 

StDev 

SE 

Mean 

Flight  #2  50 

-83.50 

3.21 

0.45 

Setup  #1  50 

-83.92 

1.90 

0.27 

Difference  =  mu  (Flight 
Estimate  for  difference: 
95%  Cl  for  difference: 

#2)  -  mu 
0.425 
(-0.624, 

(Setup 

1.474) 

#1) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  0.81  P-Value  =  0.423  DF  =  79 
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Two-Sample  T-Test  and  Cl:  Flight  #2,  Setup  #2  (Snapshot) 


N 

Mean 

StDev 

SE  Mean 

Flight  #2 

50 

-83.50 

3.21 

0.45 

Setup  2 

50 

-88.81 

1.90 

0.27 

Difference  =  mu 

(Flight 

#2)  -  mu 

(Setup 

#2) 

Estimate  for  difference:  5.306 

95%  Cl  for  difference:  (4.257,  6.355) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  10.07  P-Value  =  0.000  DF  =  79 


Two-Sample  T-Test  and  Cl:  Flight  #2,  Isotropic  Antenna  (Snapshot) 

N  Mean  StDev  SE  Mean 

Flight  #2  50  -83.50  3.21  0.45 

Isotropic  Antenna  50  -64.736  0.712  0.10 

Difference  =  mu  (Flight  #2)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -18.764 

95%  Cl  for  difference:  (-19.697,  -17.832) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -40.37  P-Value  =  0.000  DF  =  53 


Two- Sample 

T-Test  and  Cl 

:  Flight 

#2,  Setup  #1  (Throughput) 

N 

Mean 

StDev  SE 

Mean 

Flight  #2 

3674 

40030 

26686 

440 

Setup  #1 

3674 

39236 

11331 

187 

Difference  = 

mu  (Flight 

#2)  - 

mu  (Setup 

#1) 

Estimate  for 

difference : 

:  794 

95%  Cl  for  difference: 

(-144, 

1732) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  1.66  P-Value  =  0.097  DF  =  4955 


Two-Sample  T-Test  and  Cl :  Flight  #2 ,  Setup  #2  (Throughput) 

N  Mean  StDev  SE  Mean 
Flight  #2  3674  40030  26686  440 

Setup  #2  3674  41191  6058  100 

Difference  =  mu  (Flight  #2)  -  mu  (Setup_2) 

Estimate  for  difference:  -1161 
95%  Cl  for  difference:  (-2047,  -276) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -2.57  P-Value  =  0.010  DF  = 
4050 


Two-Sample  T-Test  and  Cl:  Flight  #2,  Isotropic  Antenna  (Throughput) 

N  Mean  StDev  SE  Mean 
Flight  #2  3674  40030  26686  440 
Isotropic  Antenna  3674  42385  4593  76 


Difference  =  mu  (Flight  #2)  -  mu  (Isotropic  Antenna) 
Estimate  for  difference:  -2355 

77 


95%  Cl  for  difference:  (-3231,  -1479) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -5.27  P-Value  =  0.000  DF  = 
3890 


B.3.  Flight  #3  Analysis 

Two-Sample  T-Test  and  Cl:  Flight  #3,  Setup  #1 

N  Mean  StDev  SE  Mean 
Flight  #3  4505  -92.02  5.80  0.086 

Setup  #1  4505  -90.95  7.04  0.10 

Difference  =  mu  (Flight  #3)  -  mu  (Setup  #1) 

Estimate  for  difference:  -1.069 

95%  Cl  for  difference:  (-1.336,  -0.803) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -7.87  P-Value  =  0.000  DF  = 
8688 


Two-Sample  T-Test  and  Cl:  Flight  #3,  Setup  #2 


N 

Mean 

StDev 

SE  Mean 

Flight  #3 

4505 

-92.02 

5.80 

0.086 

Setup  #2 

4505 

-93.99 

6.22 

0.093 

Difference  =  mu 

(Flight 

#3)  -  mu 

(Setup 

#2) 

Estimate  for  difference:  1.975 

95%  Cl  for  difference:  (1.727,  2.224) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  15.59  P-Value  =  0.000  DF  = 
8964 


Two-Sample  T-Test  and  Cl:  Flight  #3,  Isotropic  Antenna 

N  Mean  StDev  SE  Mean 
Flight  #3  4505  -92.02  5.80  0.086 

Isotropic  Antenna  4505  -71.09  5.15  0.077 

Difference  =  mu  (Flight  #3)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -20.933 

95%  Cl  for  difference:  (-21.159,  -20.706) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -181.16  P-Value  =  0.000  DF  = 
8884 


Two-Sample  T-Test  and  Cl:  Flight  #3,  Setup  #1  (Snapshot) 


N 

Mean 

StDev 

SE  Mean 

Flight  #3 

50 

-74 .96 

3.05 

0.43 

Setup  1 

50 

-77 .45 

6.33 

0.90 

Difference  =  mu 

(Flight 

#3)  -  mu 

(Setup 

#1) 

Estimate  for  difference:  2.490 

95%  Cl  for  difference:  (0.508,  4.473) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  2.51  P-Value  =  0.015  DF  =  70 
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Two-Sample  T-Test  and  Cl:  Flight  #3,  Setup  #2  (Snapshot) 


N  Mean  StDev  SE  Mean 

Flight  #3  50  -74.96  3.05  0.43 

Setup  #2  50  -76.820  0.225  0.032 

Difference  =  mu  (Flight  #3)  -  mu  (Setup  #2) 

Estimate  for  difference:  1.860 

95%  Cl  for  difference:  (0.990,  2.729) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  4.30  P-Value  =  0.000  DF  =  49 

Two-Sample  T-Test  and  Cl:  Flight  #3,  Isotropic  Antenna  (Snapshot) 

N  Mean  StDev  SE  Mean 

Flight  #3  50  -74.96  3.05  0.43 

Isotropic  Antenna  50  -53.815  0.226  0.032 

Difference  =  mu  (Flight  #3)  -  mu  (Isotropic  Antenna) 

Estimate  for  difference:  -21.145 

95%  Cl  for  difference:  (-22.014,  -20.275) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -48.88  P-Value  =  0.000  DF  =  49 

Two-Sample  T-Test  and  Cl :  Flight  #3 ,  Setup  #1  (Throughput) 


N 

Mean 

StDev 

SE 

Mean 

Flight  #3  4505 

52959 

20498 

305 

Setup  #1  4505 

52880 

13156 

196 

Difference  =  mu  (Flight  #3) 

-  mu 

(Setup 

#1) 

Estimate  for  difference:  80 

95%  Cl  for  difference:  (-632,  791) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  0.22  P-Value  =  0.826  DF  =  7676 


Two-Sample  T-Test  and  Cl:  Flight  #3,  Setup  #2  (Throughput) 

N  Mean  StDev  SE  Mean 
Flight  #3  4505  52959  20498  305 

Setup  #2  4505  55059  7635  114 

Difference  =  mu  (Flight  #3)  -  mu  (Setup  #2) 

Estimate  for  difference:  -2100 

95%  Cl  for  difference:  (-2739,  -1461) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -6.44  P-Value  =  0.000  DF  = 
5730 

Two-Sample  T-Test  and  Cl:  Flight  #3,  Isotropic  Antenna  (Throughput) 

N  Mean  StDev  SE  Mean 
Flight  #3  4505  52959  20498  305 

Isotropic  Antenna  4505  56050  4562  68 

Difference  =  mu  (Flight_Test_3)  -  mu  (Isotropic) 

Estimate  for  difference:  -3091 

95%  Cl  for  difference:  (-3704,  -2477) 

T-Test  of  difference  =  0  (vs  not  =) :  T-Value  =  -9.88  P-Value  =  0.00  DF  = 
4949 
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Appendix  C.  Supporting  Figures 


This  appendix  contains  Flight  #2  and  Flight  #3  figures  to  support  the  analysis  of 
this  study.  These  support  discussion  in  section  4.3. 


C.l.  Flight  #2 


Figure  30:  Comparison  of  RSSI  values  for  Flight  #2  vs  Antenna  setup  #1 


Time  Series  Plot  of  Test_  Flight_  #2  vs  Setup_  #2 


Time  Series  Plot  of  Test_Flight_#2  vs  Setup_#2  (Snapshot) 


Figure  31:  Comparison  of  RSSI  values  for  Flight  #2  vs  Antenna  setup  #1 


Figure  32:  Comparison  of  RSSI  values  for  Flight  #2  vs  Isotropic  Antenna 
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Scatterplot  of  RSSI  vs  Distance  (Test_  Flight_  #2,  Setup_ #1) 
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Distance  (meters) 


Figure  33:  Scatterplot  of  Distance  vs  RSSI  (Flight  #2,  Setup  #1) 


Scatterplot  of  RSSI  vs  Distance  (Test_  Flight_  #2,  Setup_  #2) 
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Figure  34:  Scatterplot  of  Distance  vs  RSSI  (Flight  #2,  Setup  #2) 


Scatterplot  of  RSSI  vs  Distance  (Test_Flight_#l,  I  sotropic_  Antenna) 
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Figure  35:  Scatterplot  of  Distance  vs  RSSI  (Flight  #2,  Isotropic) 
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Scatterplot  of  Throughput  vs  Distance  (Flight  #2,  Setup  #1) 
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Figure  36:  Scatterplot  of  Flight  #2  vs  Setup  #1 


Scatterplot  of  Throughput  vs  Distance  (Flight  #2,  Setup  #2) 
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Figure  37:  Scatterplot  of  Flight  #2  vs  Setup  #2 


Scatterplot  of  Throughput  vs  Distance  ( Flight  #2, 1  sotropic  Antenna) 
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Figure  38:  Scatterplot  of  Flight  #2  vs  Isotropic 
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C.2.  Flight  #3 


Figure  39:  Comparison  of  RSSI  values  for  Flight  #3  vs  Setup  #1 


Figure  40:  Comparison  of  RSSI  values  for  Flight  #3  vs  Setup  #2 


Time  Series  Plot  of  Test_  Flight_  #3  vs  I  sotropic_  Antenna 


2 


Variable 

-  Test_Flight_#3 
■  lsotropic Antenr 


Time  Series  Plot  of  Test_  Flight_  #3  vs  I  sotropic_  Antenna 


-  Test_Flight_#3 
■  lsotropic Antenra 


Figure  41:  Comparison  of  RSSI  values  for  Flight  #3  vs  Isotropic  Antenna 
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Figure  42:  Scatterplot  of  Distance  vs  RSSI  (Flight  #3,  Setup  #1) 


Scatterplot  of  RSSI  vs  Distance  (Test_Flight_#3,  Setup_#2) 
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Figure  43:  Scatterplot  of  Distance  vs  RSSI  (Flight  #3,  Setup  #2) 


Figure  44:  Scatterplot  of  Distance  vs  RSSI  (Flight  #3,  Isotropic) 
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Scatterplot  of  Throughput  vs  Distance  (Flight  #3,  Setup  #1) 
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Figure  45:  Scatterplot  of  Flight  #3  vs  Setup  #1 


Scatterplot  of  Throughput  vs  Distance  (Flight  #3,  Setup  #2) 
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Figure  46:  Scatterplot  of  Flight  #3  vs  Setup  #2 


Scatterplot  of  Throughput  vs  Distance  ( Flight  #3, 1  sotropic  Antenna) 
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Figure  47:  Scatterplot  of  Flight  #3  vs  Isotropic 
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Appendix  D.  Flight  Paths 


This  appendix  contains  the  flight  path  trajectory  graphs  for  flight  #2  and  flight  #3. 


D.l.  Flight  #2 


Figure  48:  Flight  Path  Trajectory  for  Flight  #2 


D.2.  Flight  #3 


Figure  49:  Flight  Path  Trajectory  for  Flight  #3 
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validated  network  model  that  emulates  UAV  network  behavior  during  flight,  using  a  leading  simulation  tool.  Ultimately,  there  is  a 
great  need  for  a  simulation  environment  that  provides  the  capability  to  evaluate  several  aspects  of  networked  UAVs,  in  lieu  of  large 
test-beds  or  costly  flight  testing.  These  simulations  are  designed  to  understand  the  characteristics  and  essential  performance 
parameters  of  the  delivered  model.  A  statistical  analysis  is  performed  to  explain  results  obtained,  and  identify  potential  performance 
irregularities.  A  systemic  approach  is  taken  during  the  preparation  simulation  phase  to  avoid  producing  misleading  results. 
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